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Executive  Summary 

In  May  2007  General  Dynamics  Information  Technology  was  awarded  Contract  N00014-07- 
C-0144  “Validation  and  Implementation  of  Sensor  Sweet  Spot  Selection  Algorithms”  by  the 
Office  Of  Naval  Research  (ONR)  to  develop  a  process  to  identify  and  quantify  the  tracking 
performance  of  an  existing  passive  acoustic  sensor  subsystem  (AN/BSY-2  Wide  Aperture  Array 
(WAA)),  for  contacts  characterized  by  various  received  sensor  parameters  within  the  sensor’s 
performance  envelope.  As  a  result  of  the  analysis  of  those  parameters,  we  proposed  to  develop 
quantifiable  weighting  factors;  such  that  the  sonar  sensor’s  performance  can  be  characterized 
based  on  a  contact’s  location  within  various  parametric  and  volumetric  boundaries,  defined  by  a 
set  of  ranges,  bearings,  bearing  rates,  Depression/Elevation  (D/E)  angles,  Signal-to-Noise  Ratios 
(SNR),  and  other  possible  parameters.  The  resultant  weighting  factors  associated  with  the  sensor 
data  that  is  input  to  the  various  Target  Motion  Analysis  (TMA)  processing  techniques  will 
provide  the  ship’s  force  with  a  higher  confidence  level  in  the  quality  of  the  resultant  TMA 
solutions  and  a  better  picture  of  all  contacts  within  the  ship’s  battlespace,  in  addition  to  providing 
more  rapid  and  accurate  firing  solutions  on  contacts  of  interest. 

The  focus  of  the  first  year  of  this  research  effort  was  on  data  collection,  data  definition  and 
translation,  database  creation,  the  verification  and  refinement  of  previously  defined  WAA  Sweet 
Spot  parameters,  refinement  of  the  dynamic  Sweet  Spot,  including  additional  contributions,  and 
a  demonstration  of  the  positive  effects  on  TMA  capabilities.  Due  to  the  delays  incurred 
obtaining  existing  recorded  WAA  tracker  and  developing  tools  to  correct  and/or  adjust  recorded 
data  errors,  we  were  not  able  to  conduct  a  demonstration  of  the  positive  effects  of  the  Sweet  Spot 
on  TMA  performance. 

This  Phase  I  effort  has  developed  the  process  of  gathering,  and  formatting  at-sea  collected 
data  into  a  Common  Database  (CDB),  which  allows  further  research  into  developing  weighting 
factors  that  characterize  the  quality  of  the  WAA  sensor’s  received  data  at  various  locations 
within  the  sensor’s  performance  envelope.  A  focal  point  of  this  research  effort  was  the 
establishment  of  a  well-defined  and  documented  process  for  analyzing  and  validating  the  Sweet 
Spot  criteria.  We  have  established  a  process  that  is  repeatable,  predictable  and  can  provide  us 
with  the  basis  for  achieving  verifiable  results. 

Gathering  the  at-sea  data  from  Technical  Evaluation  (TECHEVAL),  Operational  Evaluation 
(OPEYAL)  and  other  evolutions  for  submarines  with  WAA  required  extensive  assistance  from 
outside  agencies  and  the  establishment  of  the  mechanisms  required  to  transport  that  secure  data 
to  the  General  Dynamics  Information  Technology  research  facility.  In  order  to  determine  what 
portions  of  that  data  for  various  runs  would  be  cogent  to  our  analysis  required  a  virtual  bit-by-bit 
review  of  recorded  data  files  due  to  the  wide  array  of  data  formats  from  ship  to  ship  that  had  to 
be  transformed  into  our  CDB  structure.  As  a  result  of  this  research  effort  recorded  data  are  now 
standardized  and  installed  in  our  CDB.  While  not  all  data  is  100  percent  compatible,  methods 
have  been  defined  to  maximize  the  usefulness  of  all  recorded  runs. 

Numerous  special  tools  had  to  be  developed  to  correct  and/or  supplement  errors  and  /or 
shortcomings  within  some  of  the  data  content.  These  included  data  deficiency  corrections  for 
incomplete  or  missing  pertinent  data,  environmental  data,  sound  path  correction,  and  non- 
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measurement  error  correction.  A  special  tool,  driven  by  data  extracted  utilizing  capabilities 
implemented  in  the  CDB,  has  been  developed  to  assess  the  affects  of  sound  path  and  related 
affects  caused  by  deviations  in  D/E.  Special  tools  and  techniques  have  been  established  for 
defining  and  correcting  what  are  considered  to  be  non-measurement  errors  which  result  from 
conditions  created  by  the  data  recording  process  and  those  which  become  a  factor  because  of  the 
increased  accuracy  of  Sweet  Spot  data  (e.g.,  sound  path  vice  straight  line,  installed  array  offsets, 
etc.). 

A  user-friendly  capability  to  query,  using  a  full  complement  of  relational  statements,  has 
been  designed  into  the  CDB.  This  is  the  primary  method  for  identifying  segments  of  data  which 
meet  defined  interim  Sweet  Spot  criteria  and  provides  a  standardized  method  to  review  and 
analyze  a  large  quantity  of  data  runs  containing  a  variety  of  operating  and  environmental 
conditions.  Additionally,  a  vast  array  of  flexible  and  Sweet  Spot  analysis  driven  chart  generation 
capabilities  have  been  integrated  into  the  CDB  design  with  user-friendly  chart  manipulation 
capabilities,  allowing  the  analyst  to  easily  alter  the  presentation  to  focus  on  a  particular  area  of 
concern.  The  chart  generation  capabilities  are  supported  by  capabilities  to  provide  statistical 
quantification  of  data  contained  in  the  selected  data  segment  (variance,  standard  deviation, 
average,  etc.)  There  is  an  additional  capability  to  place  selected  data  presentations  in  a 
composite  presentation,  which  greatly  enhances  analysis  efforts. 

The  capability  to  perform  a  TMA  assessment  of  Sweet  Spot  data  has  been  installed  and 
checked  out  at  both  the  General  Dynamics  Information  Technology  Facility  and  at  the  Naval 
Undersea  Warfare  Center  Division,  Newport  (NUWCDIVNPT)  facility  in  Building  1171.  While 
the  capability  to  output  data  in  a  form  compatible  with  both  facilities  is  embedded  into  the  CDB, 
the  capability  to  properly  tag  all  data  to  define  each  data  item’s  Sweet  Spot  compliance  areas  was 
not  fully  implemented. 

Only  a  preliminary  analysis  of  selected  portions  of  the  available  data  were  able  to  be 
performed  due  to  Phase  I  resource  constraints.  However,  some  new  contributing  factors  have 
been  identified  such  as;  D/E  stability,  rate  of  change  in  both  bearing  or  range,  individual  array 
SNR  (Forward-Mid  (FM),  Mid-Aft  (MA)  and  D/E)  vice  and  establish  single  composite  array 
SNR,  etc. 

The  capability  now  exists  to  perform  a  more  detailed  assessment  of  how  Sweet  Spot  data  can 
improve  TMA  performance  with  future  at  sea  data,  and  we  believe  that  additional  data  from  USS 
Virginia  TECH/OPEVAE  as  well  as  from  an  exercise  in  the  Pacific  currently  being  planned  to 
evaluate  onboard  acoustic  sensors  on  USS  Cheyenne,  could  provide  the  additional  data  needed. 
We  believe  that  since  the  process  we  have  developed  has  been  well  defined  and  verified,  an 
efficient  iterative  Sweet  Spot  assessment  of  a  statistically  significant  number  of  data  sets  could 
be  completed  to  verify,  further  refine,  and  validate  the  end  product  (i.e.,  Target  Solution)  when 
employing  the  Sweet  Spot  criteria. 
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1.0  Background 

In  May  2007  General  Dynamics  Information  Technology  was  awarded  Contract  N00014-07- 
C-0144  “Validation  and  Implementation  of  Sensor  Sweet  Spot  Selection  Algorithms”  by  the 
ONR  to  develop  a  process  to  identify  and  quantify  the  tracking  performance  of  an  existing 
passive  acoustic  sensor  subsystem,  for  contacts  characterized  by  various  received  sensor 
parameters  within  the  sensor’s  performance  envelope.  As  a  result  of  the  analysis  of  those 
parameters,  we  proposed  to  develop  quantifiable  weighting  factors;  such  that  the  sonar  sensor’s 
performance  can  be  characterized  based  on  a  contact’s  location  within  various  parametric  and 
volumetric  boundaries,  defined  by  a  set  of  ranges,  bearings,  bearing  rates,  D/E  angles,  SNR,  and 
other  possible  parameters.  The  resultant  weighting  factors  associated  with  the  sensor  data  that  is 
input  to  the  various  TMA  processing  techniques  will  provide  the  ship’s  force  with  a  higher 
confidence  level  in  the  quality  of  the  resultant  TMA  solutions  and  a  better  picture  of  all  contacts 
within  the  ship’s  battlespace,  in  addition  to  providing  more  rapid  and  accurate  firing  solutions  on 
contacts  of  interest. 

This  effort  was  a  result  of  an  analysis,  conducted  by  NUWCDIVNPT  in  FY03,  of  the 
AN/BSY-2  WAA  performance  during  the  AN/BSY-2  OPEVAL.  Specifically,  a  question  arose 
regarding  the  performance  of  the  AN/BSY-2  WAA.  That  analysis  indicated  the  capability  for 
superior  WAA  TMA  accuracy  (i.e.,  a  Sweet  Spot  in  that  sensor’s  performance  envelope)  when 
certain  basic  sensor  parameters  meet  specified  criteria.  It  was  postulated  that  the  accuracy  and 
time  to  a  firing  solution  could  have  been  greatly  enhanced  had  the  TMA  operators  been 
cognizant  of,  and  could  more  effectively  exploit,  the  observed  specific  operational  criteria 
associated  with  the  Sweet  Spot. 

The  WAA  sonar  is  a  highly  sensitive  three-dimensional  passive  ranging  sonar  system  that 
provides  direct  measurements  of  contact  bearing,  D/E  angle,  and  range.  The  WAA  consists  of 
six  externally  mounted  arrays  with  three  placed  on  each  side  of  a  submarine  platform  with  an 
inboard  electronics  subsystem  to  process  the  data.  Passive  ranging  is  performed  using  a 
technique  called  Wavefront  Curvature  Ranging  (WCR).  This  technique  compares  the  arrival 
times  of  a  propagating  wavefront  at  three  different  points  (array  centers)  in  a  line  and  determines 
the  curvature  from  the  differences  in  arrival  time.  Once  the  curvature  is  known,  the  range  from 
the  signal  source  can  be  calculated.  Target  range  and  target  bearing  are  then  passed  along  to  the 
TMA  subsystem  for  further  analysis  of  system  solution  (range,  course  and  speed). 

Previous  analysis  of  many  sets  of  WAA  sea  test  data  has  indicated  that  Wave  Front 
Curvature  (WFC)  range  and  tracker  bearing  accuracy  are  heavily  dependent  upon  several 
parameters.  The  most  important  of  these  parameters  are  SNR,  target  relative  bearing  and  target 
range.  At-sea  observations  as  well  as  theory  have  shown  that  range  accuracy  is  best  at  high 
SNRs,  target  relative  bearing  around  broadside,  and  short  ranges.  Despite  these  general  known 
conditions,  the  impacts  of  their  dynamic  details  have  not  been  defined  with  tracker  quality 
(particularly  range)  properly  tagged,  and  current  TMA  algorithms  have  not  incorporated 
meaningful  weighting  of  values  for  bearing  and  range. 
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This  Phase  I  effort  was  a  follow  on  to  the  above  analysis  to  develop  weighting  factors  that 
characterize  the  quality  of  the  WAA  sensor’s  received  data  at  various  locations  within  the 
sensor’s  performance  envelope. 

This  effort  involved  extensive  reviews  and  analysis  of  TECHEVAL  and  OPEVAL  reports 
and  other  at-sea  recorded  data  on  WAA  sensor  performance.  The  effort  has  defined  a  detailed 
technical  approach  for  locating,  collecting,  interpreting,  formatting  and  restructuring  the  data  into 
a  CDB.  It  also  details  the  iterative  relational  analysis  process  utilized  to  establish  predictable 
Sweet  Spot  contributors  and  to  assess  the  positive  affects  of  the  resulting  Sweet  Spot  tagged  data. 
It  is  expected  that  the  output  of  this  research  may  trigger  other  Sweet  Spot  study  efforts  to 
dynamically  demonstrate  the  resulting  positive  effect  on  contact  solution  generation  capabilities. 

1.1  Initial  Investigation 

In  FY03,  during  a  Senate  Armed  Services  Committee  (SASC)  staffer  briefing  on  the  Director 
of  Operational  Test  &  Evaluation  (DOT&E)  Report  on  the  AN/BSY-2  OPEVAL  results,  a 
question  arose  regarding  the  performance  of  the  AN/BSY-2  WAA.  As  a  result,  PMS-350  posed 
the  following  question  to  NUWCDIVNPT:  “Does  the  cost  of  the  WAA  justify  the  expense  when 
it  has  such  a  large  range  error  and  what  does  it  do  to  improve  the  performance  of  the  ship?” 
PMS-350  requested  a  white  paper  that  addressed  the  DOT&E  identified  range  error  deficiency, 
the  cost  versus  benefit  addressed  by  the  SASC  staffer,  and  the  impact  or  pros/cons  of  doing  a 
WAA  range  calibration. 

In  order  to  resolve  the  questions  related  to  the  excessive  AN/BSY-2  TMA  Time  of  Fire 
(TOF)  range  errors  attributed  to  the  WAA  in  that  report,  NUWCDIVNPT  conducted  an 
exhaustive  review  and  analysis  of  all  sensor  measurement  data  collected  during  the  runs  outlined 
in  the  DOT&E  report. 

1.2  Findings 

The  conditions  listed  below  summarize  the  findings  from  the  resulting  NUWCDIVNPT 
FY03  analysis.  They  identified  the  major  contributors  (relative  bearing,  range,  SNR),  but  were 
inconclusive  in  defining  the  contributions  the  lesser  known  influences  like  D/E 
tracking/multipath,  environmental  conditions,  etc.  and  the  dynamic,  interactive  application  of  all 
these  conditions.  The  process  established  for  the  research  analysis,  conducted  during  the  Sensor 
Sweet  Spot  effort  documented  in  this  report,  was  geared  toward  providing  visibility  into  the 
contributions  of  these  known  elements  as  well  as  attempting  to  determine  and  characterize  other 
possible  contributors. 

The  lessons  learned  from  the  NUWCDIVNPT  FY03  analysis  were  as  follows: 

1 .  The  WAA  sensor  is  able  to  process  bearing  and  range  data  over  a  major  portion  of  the 
ship’s  360-degree  azimuth.  However,  to  achieve  the  originally  specified  WAA  sensor 
performance,  the  target  must  be  within  much  more  limited  azimuthal  sectors  that  are 
centered  broadside  relative  to  Ownship.  Therefore,  WAA  sensor  data  processed  outside 
those  specified  target  azimuthal  bands  should  be  weighed  much  less  by  TMA  than  data 
collected  within  the  specified  bands  if  the  TMA  solution  development  accuracy  is  to  be 
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optimized.  There  is  currently  little  evidence  this  strategy  is  currently  tactically  employed 
during  TMA  solution  development  today. 

2.  It  is  required  that  the  WAA  tracker  be  allowed  to  stabilize,  subsequent  to  assignment  on 
target  trace  and  upon  entry  into  prime  Sweet  Spot  areas.  The  data  collected  prior  to 
stabilization  should  be  considered  inaccurate  and  the  TMA  function  should  weigh  the 
WAA  data  before  the  tracker  is  stabilized  much  less  than  data  collected  after  tracker 
stabilization.  If  this  rule  is  not  adhered  to,  TMA  solution  development  will  not  be 
optimized. 

3.  Although  WAA  sensor  bearing  and  range  measurements  are  generated  over  a  wide  range 
of  target  SNR,  previous  analysis  of  sea  test  data  has  found  that  WAA  ranging  accuracy 
substantially  degrades  below  a  specific  target  SNR.  Therefore,  sensor  data  collected 
outside  this  minimum  target  SNR  should  be  weighted  much  less  than  data  collected 
above  that  specified  SNR  value  during  the  TMA  solution  development  if  the  solution 
range  accuracy  is  to  be  optimized.  There  is  little  evidence  that  variable  weighting  of 
sensor  data  is  being  employed  during  TMA  on  tactical  platforms. 

4.  WAA  sensor  ranging  accuracy  is  specified  within  a  defined  range  band.  Several  of  the 
runs  contained  range  data  collected  outside  this  range  band,  both  before  and  after  TOF. 
Data  collected  and  processed  by  TMA  outside  the  specified  WAA  sensor  range  band  was 
found  to  be  less  accurate  than  that  collected  inside  the  range  band.  Much  of  the  run  data 
processed  by  TMA  was  collected  outside  that  range  band. 

5.  TMA  was  not  designed  to  identify  all  the  above  constraints  associated  with  WAA  sensor 
performance.  For  the  most  part,  TMA  weighs  sensor  measurements  equally,  providing 
only  very  limited  functionality  to  automatically  filter  unreasonable  data.  TMA  operators 
are  trained  to  eliminate  outlying  measurements  they  consider  to  be  unreasonable,  and 
those  that  disagree  substantially  with  the  majority  of  data  collected  by  the  sensor. 
However,  there  is  no  automated  functionality  to  help  TMA  operators  identify  highly 
accurate  range  data  from  that  which  is  likely  to  be  suspect. 

6.  Because  range  data  collected  within  the  above  specified  Sweet  Spot  constraints 
oftentimes  make  up  the  minority  of  WAA  sensor  range  measurements,  TMA  operators 
process  a  substantial  amount  of  less  accurate  WAA  sensor  range  data  concurrently  with 
highly  accurate  WAA  measurements,  thereby  diluting  the  ultimate  TMA  solution 
accuracy. 

7.  Data  provided  indicates  that  time  alignment  errors  approximating  40  to  50  seconds 
existed  between  reconstruction  and  Fire  Control.  These  errors  were  present  on  the  test 
platform  for  approximately  20  to  25  percent  of  the  runs  where  time  data  was  available. 
This  implies  that  TOF  target  solution  range  associated  with  the  test  platform  is  likely 
different  than  TOF  solution  range  associated  with  reconstruction.  Therefore,  TOF  target 
reconstruction  ranges  for  some  runs,  likely  contain  substantial  range  errors.  This  fact 
calls  into  question  the  actual  errors  calculated  in  the  report,  since  those  errors  assumed 
the  reconstruction  range  to  be  highly  accurate. 

8.  For  many  of  the  runs,  time  synchronization  errors  between  the  test  platform  and 
reconstruction  were  unavailable.  This  currently  makes  it  impossible  to  determine  the 
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accuracy  of  reconstruction  error  associated  with  the  majority  of  run  data  collected  during 
the  exercise. 

In  this  report  the  term  “Sweet  Spot”  refers  to  almost  anything  that  embodies  an  optimum 
combination  of  characteristics  and  qualities,  therefore,  it  is  the  most  efficient,  useful,  popular,  or 
the  most  lucrative  entity  in  a  group.  What  is  a  “Sensor  Sweet  Spot?”  The  Sensor  Sweet  Spot 
arises  from  a  set  of  operational  and  environmental  conditions  which  produce  an  optimal  sensor 
output  quality.  These  improvements  can  manifest  themselves  in  either  basic  detection,  bearing 
tracking  or  more  sensitive  measurements  like  the  passive  ranging  features  of  the  sensor.  This 
research  effort  prominently  focused  on  tracking  capabilities  with  an  emphasis  on  the  more 
sensitive  measurements  which  will  likely  present  the  greatest  positive  effect  on  TMA 
performance. 

The  Sweet  Spot  is  very  dynamic  with  the  key  parameters  being  range,  bearing  and  SNR. 
These  factors  are  variable  on  their  own;  but  when  other  factors,  such  as  Ownship  speed,  sound 
path  conditions,  range  rate,  bearing  rate,  individual  (FM,  MA  and  D/E)  tracker  stability,  etc. 
were  added,  the  window  of  “sweet”  data  can  change  drastically.  For  example,  we  may  find  a 
Sweet  Spot  probability  area,  but  then  if  Ownship  speed  is  high,  we  may  find  no  situation  that 
meets  the  Optimum  Sweet  Spot  range  accuracy  criteria. 

If  we  look  at  just  the  basic  contributors,  we  find  that  with  higher  SNR  values  we  will  see  a 
fairly  large  area,  measured  by  relative  bearing  and  range,  meeting  the  Sweet  Spot  criteria.  The 
measured  bearings  and  ranges  within  the  Optimum  Sweet  Spot  area  will  exhibit  accuracies 
conducive  to  meaningful  solution  generation  from  greatly  reduced  data  sets.  We  notice  that  as 
the  range  and  bearing  boundary  increases,  there  is  a  lower  threshold.  When  this  occurs,  the  area 
of  “sweet”  data  is  now  lying  outside  the  best  zone.  We  can  provide  operational  guidance  for 
maneuvering  to  get  the  best  solution,  and  automatically  alter  the  solution  generation  processing 
approach  to  obtain  the  best  possible  solution  given  the  existing  data  accuracy. 

Figure  1  and  Figure  2  depict  hypothetical  presentations  of  the  Sweet  Spot  probability  areas 
for  both  high  and  low  SNRs  respectively.  As  the  SNR  decreases,  the  area  of  probability  for  good 
quality  data  Sweet  Spot  decreases.  Not  only  are  the  optimum  Sweet  Spot  and  surrounding 
useable  areas  size  reduced,  but  the  effects  of  lesser  contributing  factors  increase. 
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While  the  FY03  analysis  strongly  indicated  that  the  Sensor  Sweet  Spot  does  actually  exist, 
this  research  effort  has  attempted  to  define  the  ideal  combination  of  sensor  data  parameters, 
which  when  used  to  enhance  today’s  sensors’  algorithms,  will  provide  operators  with  a  more 
clear  and  consistent  picture  of  the  “best”  area  within  that  sensor  to  prosecute  a  contact  in  order  to 
achieve  the  most  successful  weapon  firing  or  contact  location  for  position  monitoring  and/or 
collision  avoidance. 

While  some  of  the  key  components  defining  the  Sweet  Spot  boundaries  are  not  only  obvious 
(SNR,  range,  relative  bearing),  but  have  been  proven,  in  a  general  sense,  going  all  the  way  back 
to  testing  performed  using  the  WAA  Advanced  Development  Model  (ADM),  until  now,  the 
refinement  of  these  factors  and  the  definition  of  other  considerations  which  could  produce  an 
acceptable  probability  of  the  prediction  of  when  Sweet  Spot  conditions  exist  has  been  lacking. 

A  core  objective  of  this  research  has  been  to  establish  a  process  whereby  Sweet  Spots  can  be 
determined  successfully  and  routinely  in  any  sensor.  Once  we  have  established  the  Sweet  Spot 
mechanisms  in  the  WAA,  the  usefulness  and  value  of  this  research  can  be  extended  to  other 
sensors:  Acoustic,  Electro-Optical,  Infrared,  Electromagnetic,  etc. 
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2.0  Analysis  Process  Overview 

The  focus  of  this  first  year  of  the  “Validation  and  Implementation  of  Sensor  Sweet  Spot 
Selection  Algorithms”  research  effort  was  on  data  collection,  data  definition  and  translation, 
database  creation,  the  verification  and  refinement  of  previously  defined  WAA  Sweet  Spot 
parameters,  refinement  of  the  dynamic  Sweet  Spot,  including  additional  contributions,  and  a 
demonstration  of  the  positive  effects  on  TMA  capabilities,  if  possible,  within  the  funding 
constraints  of  this  effort.  The  importance  of  a  well-defined  and  documented  process  for 
analyzing  and  proving  the  Sweet  Spot  criteria  was  just  as  crucial  and  was  the  key  focus  in  this 
first  phase,  and  we  have  established  a  process  that  is  repeatable,  predictable  and  can  provide  us 
with  the  basis  for  verifiable  results. 

The  complexity  of  that  process  was  driven  by  the  dynamic  and  diverse  nature  of  the  problem 
that  needed  to  be  solved.  While  the  basis  of  the  WAA  Sweet  Spot  was  clear  (i.e.,  a  high  signal 
strength  from  a  target  off  the  beam  with  a  reasonably  short  range  to  enhance  the  wave  front 
delineation),  these  characteristics  were  applied  to  a  review  of  existing  reports  originating  from  as 
far  back  as  the  WAA  ADM,  Weapon  System  Acceptance  Trial  (WSAT)  and  TECH/OPEVAL. 
The  results  of  this  review  are  contained  in  Section  3.3.2  and  produced  a  two  level  range  and 
single  level  bearing  Sweet  Spot  criteria  which  was  used  for  the  initial  queries  of  the  collected 
recorded  data  files. 

The  process  we  developed  utilized  WAA  recorded  data  from  past  exercises,  to  modify  and 
apply  dynamics  to  existing  data  parameters  in  order  to  identify  additional  potential  Sweet  Spot 
contributors.  The  interactive  nature  of  the  Sweet  Spot  contributing  factors  created  a  dependence 
on  a  large  sample  of  data  sets  in  order  to  accommodate  a  complete  range  of  operational 
conditions  (environmental  and  geometric).  That  search  for  applicable  historical  data  resulted  in 
data  sets  which  only  minimally  satisfied  our  analysis  needs,  therefore,  we  were  able  to  advance 
the  concept  but  not  totally  define  the  Sweet  Spot  and  all  its  associated  dynamics.  We  believe 
that  additional  data  from  USS  Virginia  TECH/OPEVAL,  and  an  exercise  in  the  Pacific  currently 
being  planned  to  evaluate  onboard  acoustic  sensors  on  USS  Cheyenne,  should  provide  the 
additional  data  needed.  While  the  data  from  these  events  was  not  available  for  analysis  during 
this  first  phase  of  this  “Validation  and  Implementation  of  Sensor  Sweet  Spot  Selection 
Algorithms”  research  effort,  we  believe  that  since  the  process  we  have  developed  is  well  defined 
and  verified,  a  quick  analysis  could  be  completed  to  further  refine  the  Sweet  Spot  criteria  when 
this  data  is  available. 

The  overall  Research  Analysis  Process  consisted  of  three  interactive/iterative  processes 
depicted  in  Figure  3: 

1 .  Raw  data  collection,  restructuring,  storage  and  non-measurement  error  adjustment  (Data 
Collection  and  Database  Creation) 

2.  Sweet  Spot  Segment  analysis  and  dynamic  conditions  definitions  (Sweet  Spot  Analysis 
Process)  and 

3.  Sweet  Spot  segment  applications  to  TMA  solution  improvement  (TMA  Assessment 
Process) 
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The  historical  data  used  for  this  analysis  was  collected  on  a  hull-by-hull  basis,  reformatted  to 
a  common  structure,  and  then  stored  in  a  database.  Because  data  analysis  history  has 
demonstrated  the  possibility  of  errors  in  recorded  data  which  may  not  apply  to  the  actual  sensor 
measurement,  we  had  to  factor  this  into  our  process.  The  analysis  process  recognized  the 
potential  impact  of  these  errors  and  made  every  effort  to  identify  their  existence  and  adjust  the 
data  accordingly  where  possible.  These  errors  are  referred  to  as  non-measurement  errors 
throughout  the  remainder  of  this  report.  All  stored  runs  were  assessed  for  these  errors  using 
existing  and  newly  developed  tools/techniques  to  identify  and  correct  or  define  a  compensating 
approach  to  reduce  the  effects  of  these  non-measurement  errors. 


Figure  3.  Overall  Process  Flow 

2.1  High-Level  Constraints  and  Assumptions 


This  analysis  effort  was  limited  to  the  WAA  installed  with  AN/BSY-2,  AN/BQG-5  and 
Virginia  Class  Combat  Systems.  It  focused  on  the  definition  of  predictable  dynamic  Sweet  Spot 
characteristics  and  the  verification  of  the  resulting  improvements  to  overall  TMA  capabilities. 
Where  possible,  existing  information  and  tools  were  used. 


Other  high-level  assumptions  were  that  there  would  be: 

•  An  adequate  availability  of  existing  recorded  WAA  tracker  data  containing  data 
elements  defining  Ownship  truth,  target  truth,  detailed  WAA  tracker  package, 
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onboard  Ownship  data,  environmental  conditions  including,  at  a  minimum, 
bottom  contributors  and  Sound  Velocity  Profile  (SVP)  and  special  conditions 
noted  during  the  recorded  exercise. 

•  Access  to  a  baseline  Combat  Control  System  that  contained  the  basic  TMA 
functions  of: 

Multi-Measurement  Performance  Evaluation  Plot  (PEP)  and  Modular 
Automatic  Test  Equipment  (MATE) 

Background  solution  processing;  i.e.,  Modified  Polar  Kalman  Statistical 
Tracker  (MPKAST)  and  Mission  Level  Evaluation  (MLE) 

•  Access  to  existing  tools  to  adjust  recorded  data  errors  resulting  from  incorrect 
truth,  erroneous  offset,  improper  environment  reconstruction  and/or  incorrect 
operator  setup  conditions,  specifically  tool(s)  to: 

adjust  all  truth  data  based  upon  defined  contact  position  error 
-  redefine  all  tracker  data  for  changes  in  array  offsets 

adjust  tracker  data  for  erroneous  operator  entry  of  local  sound  speed  when 
there  is  a  sound  head  failure  (done  automatically  otherwise) 
adjust  tracker  data  to  reflect  sound  path  vice  straight  line  characteristics 
adjust  erroneous  time  bias  recorded  and/or  truth  data. 

2.2  Data  Collection  Process  Objectives 

The  Data  Collection  Process  consisted  of  several  interactive  procedures  and  the  coordination 
of  the  efforts  of  multiple  organizations/agencies.  This  process  began  by  locating  recorded 
historical  WAA  tracking  data  and  culminated  with  a  “user-friendly”  distribution  of  criteria 
driven  data  query  results.  The  steps  developed  to  achieve  and  prove  that  process  were  developed 
to  meet  the  overall  process  goals,  not  only  between  the  interdependent  activities  within  the 
process  itself,  but  also  to  accommodate  those  contributions  required  from  other  major  process 
areas.  The  coordination  of  all  these  contributors  (internal  and  external)  was  anticipated  to  be  a 
major  effort  and  proved  to  be  more  difficult  than  expected.  The  following  is  a  list  of  the  process 
segments  defined  for  this  effort.  The  details  of  each  process  segment  are  documented  in 
Section  3. 

•  Data  Definition  and  Acquisition 

•  Environmental  Data  Definition 

•  Sound  Path  Computation 

•  Non-Measurement  Error  Update 

The  resulting  process  and  procedures  are  compatible  with  potential  future  data  sets  from  USS 
Virginia  TECH/OPEVAL  and  upcoming  USS  Cheyenne  sensor  evaluation  runs  in  the  Pacific 
when  that  data  becomes  available  to  any  follow-on  to  this  research  effort. 

2.3  Sweet  Spot  Analysis  Process  Objectives 

The  Sweet  Spot  Analysis  was  the  heart  of  this  research  project  and  consisted  of  five  major 
activities: 
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1 .  First,  charts  generated  by  the  Data  Collection  Process  were  reviewed  and  the  applicable 
WAA  and  Reference  Track  data  sets  were  defined  and  used  to  control  the  inputs  to  the 
CDB. 

2.  Once  the  data  was  populated  within  the  CDB,  the  necessary  charts  and  supporting  data 
were  selected  for  the  non-measurement  error  assessment.  When  these  error  conditions 
were  defined  and  a  data  adjustment  approach  was  developed,  the  resulting  adjustments 
were  provided  to  the  Data  Collection  Process  for  a  CDB  update.  If  no  adjustment 
capabilities  existed  or  were  needed,  the  assessment  results  along  with  any  required 
workaround  guidance  were  documented  and  incorporated  into  the  Sweet  Spot  Analysis 
data  set. 

3.  We  developed  a  tool  which  calculates  sound  paths  for  both  the  calculated  optimum  and 
the  D/E  measured  path  (including  potential  multi-paths  around  each).  The  results  of 
these  runs  were  placed  in  the  CDB  and  became  an  integral  part  of  the  path  and  D/E 
contributions  to  Sweet  Spot. 

4.  With  the  updated  and  error  reduced  data  residing  in  the  CDB,  the  Sweet  Spot  analysis 
was  able  to  begin.  The  actual  analysis  consists  of  a  combination  of  visual  presentations 
supported  by  statistical  computations.  The  desired  content  is  selected  using  a  relational 
extraction  of  the  iteratively  updated  analysis  driven  Sweet  Spot  parameters. 

5.  When  individual  runs  achieve  the  defined  level  of  Sweet  Spot  bearing  and  range 
accuracy,  they  are  passed  to  the  TMA  Assessment  Process  for  further  evaluation. 

The  key  points  of  this  analysis  process  are  that  it  provides  the  basic  groundwork  in  the 
ultimate  goal  of  establishing  the  degree  of  potential  tracker  accuracy  that  can  be  expected  from 
the  WAA  when  definable  criteria  have  been  met.  The  data  applied  and  the  analysis  process  we 
have  implemented  focus  on  the  measuring  capabilities  within  the  WAA  and  attempts  to  remove 
data  error  characteristics  reflective  of  either  recording  deficiencies  or  tracker  processing  design 
limitations.  Section  3  will  provide  additional  detail  related  to  the  specific  actions  taken  to 
achieve  the  objectives  in  each  of  the  five  major  activities  listed  above. 

2.4  TMA  Assessment  Process  Objectives 

The  TMA  contribution  to  the  Sweet  Spot  Analysis  Process  lies  in  achieving  an  overall 
effectiveness  measurement  of  the  tagged  Sweet  Spot  data.  The  brief  segments  of  qualified  run 
data  and  the  resultant  cumulative  content  of  Sweet  Spot  qualifying  data  elements  forced  us  to 
face  the  question  of  the  usefulness  of  this  data  even  when  accurately  predicted.  But  by  injecting 
identical  sets  of  tagged  and  untagged  data  and  comparing  results  in  a  controlled  TMA 
environment,  the  effects  could  be  measured  and  valuable  results  obtained.  These  results  were 
then  analyzed  and  the  findings  were  applied  in  the  manner  expressed  in  one  or  more  of  the  three 
steps  listed  below: 

1 .  The  results  reflect  a  significant  improvement  using  the  existing  Sweet  Spot  tag  criteria 
and  the  TMA  data  filtering  and  application  to  key  algorithms  produces  acceptable  results. 
This  particular  run  will  be  set  aside  and  only  be  rerun  if  other  analysis  produces  changes 
to  either  the  Sweet  Spot  tag  criteria  or  TMA  handling. 


12 

UNCLASSIFIED 


UNCLASSIFIED 


2.  The  results  do  not  indicate  an  effective  improvement  and  a  review  of  the  data  and 
possibly  associated  data  editing  indicate  a  need  to  alter  the  Sweet  Spot  criteria.  The 
related  information  will  be  transferred  to  the  Sweet  Spot  Analysis  portion  of  the  process 
and  needed  changes  evaluated. 

3.  The  results  do  not  indicate  an  effective  improvement  and  a  review  of  the  data  and 
possibly  associated  data  editing  indicate  a  need  to  alter  the  handling  of  the  Sweet  Spot 
tagged  data  with  TMA.  The  related  information  will  be  used  within  the  TMA 
Assessment  Process  portion  of  the  process  and  needed  changes  evaluated. 

Any  combination  of  the  above  results  and  actions  are  possible  and  may  be  addressed 
concurrently.  The  TMA  Assessment  Process  accommodates  the  coordination  of  all  actions  and 
the  retention  of  a  complete  record  of  data  set  (run)  status. 
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3.0  Analysis  Process  Structure 

Technical  teams  responsible  for  each  of  the  major  components  of  the  analysis  process  were 
assembled  and  their  roles  and  responsibilities  were  briefed  and  defined  in  the  Research  Plan  for 
this  effort.  The  requirements  for  this  first  phase  of  the  research  effort  were  to;  (1)  establish  a 
facility  to  handle  the  data  collected  and  house  the  CDB  and  TMA  capabilities,  (2)  collect  and 
standardize  historical  data,  (3)  create  the  CDB,  (4)  establish  the  mechanisms  to  efficiently  query 
and  analyze  the  data  to  define  and  refine  the  Sweet  Spot  conditions  and  (5)  provide  a  TMA 
capability  to  better  define  any  operational  optimizations.  These  efforts  are  partitioned  into  the 
four  listed  responsibility  areas: 

•  Facilities 

•  Data  Collection/DB  Creation  Process 

•  Sweet  Spot  Analysis  Process 

•  TMA  Assessment  Process 

Each  responsibility  area  had  a  technical  leader  who  worked  with  the  Systems  Engineering 
Technical  Lead  to  direct  and  support  the  mission  of  the  respective  teams.  Their  specific  tasks 
and  overall  analysis  approaches  are  defined  in  the  supporting  paragraphs  of  Section  3. 

3.1  Facilities 

A  facility  was  established  at  General  Dynamics  Information  Technology  in  Middletown,  RI 
to  provide  the  physical  capability  to  store  the  vast  amount  of  supporting  data,  efficiently  perform 
all  analysis  processing,  and  house  the  required  Combat  Control  TMA  baseline  TI04  capabilities. 
Additionally,  modifications  were  coordinated  with  NUWC  to  provide  two-way  data  transfer 
capabilities  between  the  General  Dynamics  Information  Technology  facility  and  NUWC.  This 
interface  was  vital  to  support  the  transfer  of  raw  data  to  General  Dynamics  Information 
Technology  and  tagged  data  to  NUWC  for  their  independent  assessment.  Figure  4  provides  the 
installation  equipment  and  software  contained  within  the  facility  and  associated  functional 
connectivity  for  the  data  collection  and  database  creation  process. 
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Figure  4.  Data  Collection  and  Database  Creation  Process 

The  Data  Collection  process  included  the  identification  of  potential  sources  and  collection  of 
applicable  recorded  data,  restructuring  that  data  into  a  CDB  and  modifying  the  data  content  to 
extract  non-measurement  error  conditions.  Figures  5,  6,  and  7  depict  the  processes  used  for 
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collecting,  restructuring,  and  storing  data  into  the  CDB  and  the  eventual  input  of  data  alterations 
deemed  necessary  because  of  data  errors  caused  by  non-measurement  factors. 

3.1.1  Data  Collection 

The  Data  Collection  phase  was  separated  into  two  parallel  paths  once  data  was  received  and 
stored  at  the  General  Dynamics  Information  Technology  Facility.  One  path  prepared  the  data  set 
for  installation  into  the  Tracker,  Ownship  Navigation  (NAV),  Target  (TGT)  Truth  and  Ownship 
Truth  segment  of  the  CDB,  while  the  other  prepared  the  data  associated  with  the  run  related 
environmental  conditions.  The  following  paragraphs  define  the  key  actions  taken  to  accomplish 
the  above  objectives. 


Figure  5.  Input  Data  Processing 


3.1.1. 1  Data  Definition  and  Acquisition 

The  historical  recorded  data  was  obtained  via  the  query  of  known  experts  in  the  various 
exercise  type  data  recording  operations.  Additionally,  in  many  cases,  these  same  personnel  were 
enlisted  to  assist  in  obtaining  data  based  upon  a  team  defined  data  requirements  list.  Much  of  the 
data  identified  had  deficiencies  when  compared  to  the  defined  requirements,  but  a  process  was 
developed  to  assess  and  apply  all  data  sets,  complete  or  not,  and  the  resulting  following  approach 
evolved. 
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The  primary  data  input  path  was  established  via  NUWC  over  the  Wide  Area  Integration 
Facility  (WAIF)  network  to  General  Dynamics  Information  Technology’s  secure  lab,  directly 
onto  a  Linux  machine  or  by  hand-carry  electronic  storage  medium,  such  as  a  CD  or  DVD.  The 
initial  objective  was  to  define  the  data  collection  end  product  CDB  format  to  drive  the 
restructuring  of  all  input  data  types  and  content.  The  initial  format  selected  was  derived  from  the 
AN/BSY-2  WAA  tracker,  Ownship  NAV  data  and  other  environmental  data  messages.  This  was 
a  logical  selection  because  AN/BSY-2  would  likely  provide  the  bulk  of  the  historical  recorded 
object  data  in  either  the  legacy  or  integrated  Acoustic  Rapid  Commercial  Off-The-Shelf  (COTS) 
Insertion  (ARCI)  form  (basically  common  message  formats).  When  the  lab  was  undergoing 
setup  activities  the  NUWC-provided  (DDSC-IS)  data  types  were  the  only  packages  available  to 
begin  construct  of  the  CDB  and  the  intermediate  processes  to  achieve  the  goal  of  an  interactive 
Graphical  User  Interface  (GUI)/CDB  that  could  ascertain  Extract,  Transform,  Load  (ETL)  and 
assimilate/analyze  heterogeneous  raw  data  sets  toward  the  definition  of  the  WAA  Sweet  Spot.  In 
particular,  MySQL,  Python,  Open  Database  Connectivity  (ODBC)/Java  Database  Connectivity 
(JDBC)  modules,  OpenOffice  2.3  and  Linux  Report  Program  Generator  (RPG)/binary  updates 
were  necessary  to  enable  the  General  Dynamics  Information  Technology  lab  to  achieve  the 
desired  results.  Once  these  updates  and  setups  were  accomplished  and  tested,  the  initial  data 
load  of  in-house  NUWC  data  began. 

The  initial  process  was  established  using  the  NUWC  provided  DDSC-IS  data  from  USS 
Virginia  Class  and  USS  Connecticut.  The  approach  was  later  adapted  to  all  input  data  types  and 
structures.  The  file  format  is  evaluated  and  the  files  are  extracted,  if  necessary.  If  the  files  are  in 
a  non-binary  format  such  as  a  spreadsheet  or  Comma  Separated  Values  (CSV),  then  one  of  the 
files  is  examined  directly  in  a  spreadsheet  program,  such  as  MS  Excel,  or  OpenOffice  Calc  on 
Linux.  Within  the  directory  structure  there  are  sometimes  support  files  that  give  an  overview  of 
the  type  of  data  received  and  the  hull,  sonar  system,  date  of  test  and  other  information.  A 
combination  of  what  is  contained  within  those  support  files  and  what  appears  to  be  inside  the 
actual  data  files  dictates  the  handling  of  the  data  files.  For  example,  if  the  files  are  for  a  data 
type  that  has  been  previously  processed,  then  that  process  will  be  used  on  the  new  data.  If  the 
format  appears  to  be  new,  then  the  data  is  evaluated  within  the  spreadsheet  application.  If  the 
data  cannot  be  opened  natively  by  a  spreadsheet  application,  then  the  supplier  of  the  data  is 
contacted  to  determine  if  a  tool  or  a  process  is  already  in  use  by  the  supplier  that  will  allow  full 
extraction  of  the  data. 

3. 1.1.2  Initial  Data  Assessment 

Working  with  key  personnel  at  the  various  data  source  sites,  the  incoming  data  was  carefully 
reviewed  and  high  level  definitions  of  required  content  actions  were  applied.  These  actions 
resulted  in  the  construction  of  run  header  data  files  containing  at  a  minimum,  the  following 
items: 

•  Exercise  Type: 

Sensor  Calibration/ Accuracy 
On  Range  Weapons 
On  Range  Free  Play 
Open  Ocean 

•  Hull  and  Combat  System  Revision 

•  Date/Time  of  Exercise  and  Individual  Run 
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•  Specific  Location  (Lat/Long)  of  Individual  Run 

•  Geometric  Conditions  including  TGT/Ownship  X,Y  Charts 

•  The  Range  of  key  measured  values: 

Relative  Bearing 
TGT  Range 
-  SNR 
D/E  Angle 
etc. 

Once  the  data  is  in  a  state  where  it  can  be  evaluated,  it  is  examined  in  detail  to  determine 
what  items  or  records  may  be  missing  in  accordance  with  requirements  contained  in  the  CDB 
Specification.  There  are  three  main  sections  of  data  for  each  data  type  as  defined  in  the  CDB: 

1.  Ownship  Navigation 

2.  Range/Truth 

3.  Tracker/Sonar 

Missing  items  must  be  assessed  and  dealt  with  for  each  section.  In  some  cases,  the  data 
supplier  will  be  able  to  supply  the  missing  pieces.  In  other  cases,  the  gaps  in  the  data  can  be 
mitigated  by  calculation,  or  substitution.  For  example,  in  one  data  set,  the  only  water-referenced 
Ownship  NAV  data  were  values  for  Ownship  heading.  All  other  key  water  referenced  parameters 
were  generated  using  available  ground  reference  values  and  then  compared  and  adjusted  using 
historical  ocean  conditions.  In  this  case,  some  error  injection  was  anticipated  and  affected  data 
items  were  appropriately  tagged. 

The  Functional  Analysis  database,  which  is  a  NUWC  defined  database  structure,  was  initially 
used  to  extract  DDCS-IS  data  from  SQL  files  that  contained  over  180  separate  data  tables.  We 
used  this  as  the  starting  point  for  constructing  the  CDB  and  used  the  AN/BSY-2  to  ARCI 
software  interface  document  to  append  data  elements  that  would  be  needed  to  perform  Sweet 
Spot  analysis.  This  process  was  implemented  as  follows: 

1 .  We  used  the  source  DDCS-IS  SQL  files  as  is  to  create  a  TRUE  raw  representation  of  the 
NUWC  provided  data.  This  meant  that  all  180  data  tables  were  extracted  to  a  MySQL 
database.  It  can  be  assumed  that  this  is  an  intermediate  database  of  which  a  second  step, 
Python  script  named  fill_database.py,  would  extract  NUWC  TMA  data  into  their 
FunctionalAnalysis  DB.  Start  and  Stop  dates  in  Universal  Time  Coordinated  (UTC)  are 
input  parameters  to  this  Python  script.  This  was  accomplished  and  there  exists  a  database 
called  Functional  Analysis  in  General  Dynamics  Information  Technology’s  MySQL 
database.  This  format  was  deemed  important  to  keep/use  so  that  General  Dynamics 
Information  Technology  could  replicate  modified  Sweet  Spot  data  back  to  the  NUWC 
TMA  standard  if  required. 

a.  Upon  analyzing  the  fill  database.py  script,  we  realized  that  there  were  6  to  10  of  the 

1 80  raw  data  tables  that  were  being  used,  and  that  only  a  fraction  of  the  data  elements 
(fields)  within  those  tables  were  being  used. 

b.  Upon  further  analysis  of  these  6  to  10  data  tables,  we  realized  that  4  of  these 
(OwnshipData  and  WAATrackerStatus  along  with  cvRContactEventProperties  and 
cvRContactReportProperties)  nearly  matched  the  AN/BSY-2  to  ARCI  Software 
Interface  document  message  blocks.  Therefore,  we  maintained  that  using  these 
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DDCS-IS  data  definitions  for  the  CDB  OwnshipNav  and  CDB  Tracker  tables  ensured 
that  all  possible/probable  data  elements  would  be  defined  and  eligible  for  use  in  the 
Sweet  Spot  CDB  analysis. 

2.  The  original  AN/BSY-2  OPEVAL  data  was  now  available  and  restructured,  and  in  some 
areas  reconstructed  into  this  new  format.  Data  received  from  exercises  recorded  on  the 
Atlantic  Undersea  Test  &  Evaluation  Center  (AUTEC)  Range  was  still  undergoing  setup 
for  the  extraction  phase  and  once  ready,  was  set  up  for  the  transformation  phase  based  on 
this  new,  improved  and  enhanced  data  capture  database  structure. 

3.  Truth  data  for  NUWC-provided  data  was  still  missing  at  this  point.  NUWC 
communicated  that  DDCS-IS  data  included  TSUNAMI- AUTEC  Pargos  files  that 
contained  truth  data  for  all  event  runs.  Also  provided  were  technical  support  and  tools 
including  the  Range  Truth  Reader  (and  supporting  files  like  math.py)  which  populated 
the  tblTruth  and  tblContactTruth  tables  in  the  FunctionalAnalysis  MySQL  database. 
Minor  data  manipulation  was  then  executed  to  append  this  data  to  the  CDB  Truth  tables. 

4.  Any  DDCS-IS  data  that  may  be  provided  to  the  General  Dynamics  Information 
Technology  Sweet  Spot  project  would  now  be  easily  ETL’d  based  on  the  process  that 
manifested  from  this  data  type. 

To  further  enhance  and  automate  the  Initial  Assessment  Process  and  provide  a  means  to 
rapidly  review  data  content  during  the  Analysis  Process,  the  following  tool  was  developed.  The 
CDB  was  designed  around  the  premise  that  it  will  house  a  large  quantity  of  very  diverse  data 
sets.  Because  of  the  diverse  nature  of  the  data  and  the  various  formats  in  which  the  data  arrives, 
an  analyst  may  not  necessarily  have  a  preconceived  idea  of  where  to  begin.  One  tool  that  the 
CDB  provides  is  an  enhanced  charting  capability.  However,  when  the  CDB  is  filled  to  its 
designed  potential,  it  may  be  far  too  cumbersome  for  the  Sweet  Spot  analyst  to  examine 
hundreds  of  charts  to  find  a  good  candidate  for  evaluating  the  Sweet  Spot.  Therefore,  the  CDB 
provides  a  tool  which  gives  the  analyst  a  high-level,  tabular  summary  of  all  of  the  data  within  a 
single  event,  for  all  runs  and  all  WAA  trackers.  This  capability  is  referred  to  as  “Statistics” 
within  the  CDB.  The  analyst  selects  the  event  and  is  able  to  define  a  window  of  desired  results 
for  WAA  SNR,  WAA  DE  and  truth  range.  The  goal  is  to  allow  the  analyst  to,  for  example, 
reject  data  where  the  WAA  SNR  is  below  or  above  a  certain  threshold,  and  the  same  idea  applies 
to  WAA  DE.  Truth  range  is  included  in  case  the  analyst  wants  to  exclude  data  that  is  close 
range,  far  range  or  both.  Each  of  these  criteria  is  combined  with  a  logical  AND  statement, 
meaning  the  statistics  tool  requires  ALL  of  the  criteria  to  be  met  before  it  counts  the  data  item. 
Examples  of  representative  statistics  are  provided  in  Appendix  B,  Figure  1.  Each  line  of  data  is 
sorted  by  run,  then  contact  group,  then  WAA  tracker.  The  total  percentage  of  data  available 
under  that  definition  that  also  meets  all  of  the  criteria  previously  specified  is  shown  as  “% 

Total.”  That  percentage  is  also  broken  down  into  port-side  and  starboard-side  relative  bearings. 
The  minimum  and  maximum  relative  bearings  for  each  side  are  shown.  Next,  the  actual 
minimums  and  maximums  of  SNR,  DE  and  truth  are  shown  directly  below  the  criteria  that  were 
typed  in  by  the  user.  Finally,  a  sentence  combining  the  total  number  of  data  samples  (in 
minutes),  SNR  and  DE  average  and  standard  deviation  appears  at  the  bottom  of  each  row.  For 
any  run  in  which  contact  group  and  WAA  tracker  combination  has  no  data  meeting  the  criteria, 
no  entry  appears. 
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At  the  end  of  this  process,  if  any  gaps  remain,  the  areas  of  the  data  that  contain  those  gaps 
must  be  labeled  upon  importation  into  the  CDB.  Labeling  typically  consists  of  using  an  agreed- 
upon  value  that  lies  outside  the  expected  range  of  values;  this  place  holder  value  will  vary 
depending  on  the  data  field  in  question.  In  some  cases,  the  gaps  may  be  so  numerous  that  the 
data  set  is  not  imported  into  the  CDB  but  kept  for  the  potential  of  filling  in  the  missing  data  in 
the  future. 

At  this  point  the  process  splits  into  parallel  paths  to  address  in  one  case  the 
environmental/ocean  conditions  (LSS,  SVP,  Bottom  depth/Contour,  etc.)  and  in  the  other  case 
Ownship/Target  elements  (Tracker  Data,  Ownship  NAV  Data  and  Ownship/TGT  Truth). 

3. 1.1.3  Data  Content  Establishment 

A  number  of  the  runs  had  tracker  data  of  no  value  to  the  WAA  Sweet  Spot  analysis  and 
would  only  cause  clutter  in  both  the  database  and  analysis  presentations.  This  step  initializes  the 
process  for  elimination  of  this  excess  data.  The  primary  tool  used  was  a  truth- versus-tracker 
bearing  chart  for  all  contacts  included  in  the  run  data.  This  chart  is  constructed  and  passed  to  the 
Sweet  Spot  Analysis  Process  in  which  the  usable  content  was  defined  by  the  Subject  Matter 
Expert  (SME).  Upon  receipt  of  the  content  definition  the  specified  usable  tracker  data  sets  were 
extracted  for  further  processing. 

Since  the  current  platform  for  hosting  the  CDB  is  Microsoft  Access,  the  most  natural  way  to 
analyze  the  data  is  in  a  separate  MS  Access  database,  which  can  be  considered  as  a  “pre-CDB” 
state.  Spreadsheet  applications,  such  as  MS  Excel  and  OpenOffice  Calc,  have  a  limitation  on  the 
maximum  number  of  rows  supported  per  worksheet  (typically  about  65,000).  This  limitation  is 
not  crippling  in  every  case;  however,  it  is  usually  an  issue  when  the  data  set  received  is  not 
adequately  broken  down  into  logical  segments.  Some  data  sets  just  contain  large  quantities  of 
data,  even  after  the  pieces  have  been  grouped  into  runs.  In  all  cases  it  is  necessary  to  split  up  an 
event  into  separate  runs,  if  it  has  not  been  done  so  already.  It  is  especially  advantageous  to  do  so 
when  the  event  contains  a  large  amount  of  data.  One  logical  method  of  determining  run  break 
points  is  to  plot  the  X  and  Y  position  of  the  Ownship  and  any  targets.  Based  on  the  Ownship 
maneuvers,  runs  can  be  created  such  that  one  run  may  contain  a  meaningful  set  of  operational 
conditions  and  separate  key  potential  Sweet  Spot  contributors,  such  as  providing  a  high  rate  of 
relative  bearing  change.  In  those  cases  of  long  runs  with  high  quantities  of  data,  only  pieces  of 
the  data  set  can  be  practically  manipulated  within  a  spreadsheet  application,  and  then  the  results 
are  applied  to  the  entire  set  within  the  Access  pre-CDB. 

Each  of  the  non-flag/bit  fields  in  the  CDB  has  an  associated  and  desired  unit  of  measure.  An 
important  part  of  the  CDB  data  input  process  is  keeping  track  of  those  units.  Data  that  comes  in 
may  be  in  any  of  several  units.  The  assumption  cannot  be  made  that  the  “usual”  units  for  a  given 
piece  of  data  will  apply.  If  the  data  field  definition  is  inadequate,  or  thought  to  be  incorrect,  then 
comparative  analysis  can  usually  determine  the  unit  type.  For  example,  latitude  and  longitude 
can  be  converted  to  two-dimensional  linear  measure,  like  an  X-Y  Cartesian  plot,  or  a  rate  like 
knots,  yards/sec  and  so  on.  Latitude  and  longitude  can  also  be  used  to  calculate  and  verify  the 
course  field  by  breaking  down  by  record  each  movement  in  latitude  and  longitude  into  linear 
measure,  taking  care  to  account  for  the  linear  distance  change  as  defined  by  geographic  position. 
Then,  forming  a  right  triangle,  the  angle  is  computed.  The  length  of  the  hypotenuse  of  this 
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triangle  is  the  velocity  over  ground.  In  some  cases,  the  data  supplier  may  be  able  to  provide  a 
clarification  of  units.  One  source  of  confusion  was  determining  whether  certain  data  items  were 
referenced  to  the  ground  or  water.  Water-referenced  movement  does  not  take  into  account  the 
water  current  vector  magnitude  and  direction  (set  and  drift)  and  ground-referenced  movement 
does.  In  sorting  out  this  issue,  there  were  three  main  categories  for  truth  movement  that  could  be 
measured  in  either  ground  or  water  reference:  X-Y  position,  heading  (water)/course  (ground), 
and  speed  (water)/velocity  (ground).  There  are  many  ways  to  rectify  this  uncertainty.  Latitude 
and  longitude,  for  example,  are  referenced  to  the  earth  and,  thus,  are  ground  referenced.  Lat/long 
position  can  be  correlated  to  any  ground  referenced  measurement. 

3.1. 1.4  Data  Deficiency  Correction 

All  data  sets  were  then  reviewed  to  establish  content  and  format.  Once  the  content  was 
evaluated  and  any  “holes”  defined,  a  variety  of  procedures  for  creating  these  missing  data  items 
had  to  be  established  if  there  were  no  tools  that  existed  to  handle  those  situations.  The  following 
list  represents  a  few  of  the  major  data  deficiency  areas  addressed: 

•  No  Relative  Bearing 

•  Incomplete  Water  Reference  Ownship/Target  Data 

•  Incomplete  Ground  Reference  Ownship/Target  Data 

•  No  Ownship/Target  Truth  Data 

•  No  Tracker  Mode  indication 

•  No  Truth  Bearing  and  Range 

•  Many  Cases  of  Data  Valid  Status  Indication  Missing,  etc. 

At  this  point,  all  the  data  types  are  defined  and  missing  data  was  dealt  with  to  the  extent 
possible.  The  data  sections  of  Range/Truth  and  Tracker/Sonar  have  specific  interdependency 
relationships  that  need  to  be  defined.  Because  the  basis  of  Sensor  Sweet  Spot  is  comparing 
actual  sonar/tracker  performance  to  what  actually  happened,  each  piece  of  tracker  data  must  be 
mated  with  the  corresponding  piece  of  truth  data.  In  other  words,  the  data  that  comes  in  is  not 
necessarily  pre-correlated.  Phase  I  of  Sensor  Sweet  Spot  begins  with  this  process,  and  thus  is 
highly  reliant  on  this  process  to  be  completed  accurately.  As  such,  for  any  data  that  is  not  pre¬ 
correlated,  human  interaction  must  take  place  and  declare  the  elements  of  truth  and  tracker  data 
that  correspond  with  each  other.  This  step  requires  someone  who  is  familiar  with  sonar 
(specifically  WAA)  trackers.  For  each  piece  of  truth/range  data  that  correlates  to  one  of  the 
targets,  the  CDB  considers  that  as  a  “Contact  Group,”  but  the  concept  is  not  unlike  the  sierra 
number.  Once  each  contact  group  has  been  defined,  the  next  step  is  to  pick  out,  among  all  of  the 
available  sonar  data,  which  trackers  go  with  that  contact  group.  This  is  done  by  graphing  true 
bearing  versus  time  as  shown  in  Appendix  B,  Figures  2  through  5.  and  overlaying  the  true 
bearing  of  the  target  as  reported  by  the  range  (the  truth  data)  with  the  actual  tracker  data,  which 
comes  out  of  the  sonar  system.  When  more  than  one  target  is  present  in  a  test,  and  their  true 
bearings  are  relatively  close  or  crossing,  trackers  may  switch  targets  and  thus  will  have  to  be  re¬ 
allocated  to  the  appropriate  Contact  Group. 

The  amount  of  time  a  tracker  is  on  one  target  must  be  analyzed  and  thus  incorporated  into  the 
CDB.  That  way,  when  Sweet  Spot  analysis  begins,  the  analyst  can  call  up  a  contact  group  and 
all  associated  trackers,  without  having  to  look  through  a  lot  of  data  that  pertains  to  another 
contact  group  (or  no  contact  group  as  is  the  case  when  sonar  tracks  a  commercial  vessel).  This 
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process  is  done  for  WAA  trackers  and,  when  possible,  one  tracker  of  another  sonar  type 
(typically  the  Sphere),  is  selected  and  correlated.  This  non-WAA  tracker  is  referred  to  as  a 
reference  tracker  and  used  primarily  to  evaluate  sensor  bias  conditions.  When  all  WAA  trackers 
have  been  correlated  to  contact  groups  and  if  a  reference  tracker  has  been  assigned,  all  remaining 
sonar  data  is  marked  as  unassociated.  It  is  not  discarded  but  does  not  appear  on  any  graphs  and 
is  not  used  in  any  way  by  the  CDB.  It  remains  available  for  a  potential  future  use.  All  tracker 
assigning  and  time  cropping  is  done  in  the  pre-CDB  realm;  however,  CDB  has  the  capability  to 
modify  the  groupings  and  time  cropping  in  a  user-friendly  way. 

3. 1.1.5  Final  Data  Translation 

The  data  fdes  which  have  received  the  processing  defined  in  Paragraphs  3. 2. 1.1  through 
3.2. 1.4  were  then  translated  to  the  CDB  format  and  all  compatible  fields  installed  into  the  CDB. 
All  CDB  fields  which  we  were  unable  to  value  were  properly  identified  in  the  CDB  such  that  any 
attempts  to  access  that  data  will  produce  an  error  condition  to  the  request  initiator. 

3.1.2  Environmental  Data  Definition 

Although  not  specifically  defined  as  a  Sweet  Spot  contributor  in  the  initial  definition,  the 
operating  environment  can  often  be  a  core  contributor  to  Sweet  Spot  predictions  and  qualifying 
data  quality.  The  analysis  process  will  rely  on  the  best  possible  definition  of  the  environment 
and  bottom  conditions.  Many  of  the  incoming  data  sets  do  not  have  a  complete  recorded 
package  of  needed  data,  but  when  these  deficiencies  exist,  historical  SVP  and  Bottom  Contour 
information  were  inserted. 

3. 1.2.1  Environmental  Data  Assessment 

Although  every  effort  was  made  to  define  and  acquire  complete  environmental  and  bottom 
contour  data  content  established  during  the  exercise,  reality  has  proven  that  in  many  cases  a 
complete  package  was  not  provided.  Therefore,  the  extraction  from  the  historical  database 
option  will  be  the  most  common  approach.  The  following  actions  have  been  established  to 
handle  all  data  availability  conditions  plus  provide  some  “sanity”  checks  to  increase  confidence 
in  the  resulting  selected  data  elements: 

•  Interrogate  the  source-supplied  data  set  and  identify  all  exercise 
measured/recorded  values  for  LSS,  SVP,  Bottom  Depth  and  Bottom  Contour 

•  If  SVP  data  exist,  extract  the  files  in  place  in  temporary  store  with  the 
associated  Lat/Long  and  Date/Time 

•  If  Bottom  Depth  recorded  values  exist,  place  into  added  field  of  the  Ownship 
NAV  File  and  set  aside  for  later  sanity  check 

•  Extract  Historical  SVP  and  Bottom  Contour  information  from  the  Historical 
Database 

•  If  recorded  sensor  values  of  LSS  exist,  compare  each  sensor  value  to  the  interim 
selected  SVP  and  resolve  issues  (sanity  check) 

•  Interrogate  the  LSS  contained  in  the  WAA  Tracker  File,  compare  to  the  selected 
LSS  and  resolve  issues  (sanity  check) 

•  Compare  Bottom  Depth  measurements  to  Historical  Data  and  resolve  issues 
(sanity  check) 
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3. 1.2.2  Environmental  Data  CDB  Insertion 

Environmental  and  bottom  data  was  installed  in  the  CDB  in  a  grid  oriented  form.  Each  run 
within  every  exercise  defined  the  Lat/Long  boundaries  and  the  appropriate  grid  was  then  placed 
into  the  CDB.  The  Historical  Database  was  accessed  and  the  bottom  contour  for  the  Lat/Long 
Grid  was  extracted  and  also  stored.  At  this  point  the  measured  bottom  depth  (if  present)  was 
compared  to  the  contour  conditions  installed.  Finally,  the  SVP  characteristics  distributed  over 
the  defined  grid  were  evaluated,  and  needed  boundaries  for  SVP  change  were  established.  The 
necessary  SVPs  (historical  or  measured  as  defined  in  3.2.2. 1)  were  then  installed  in  the  CDB. 

3.1.3  Sound  Path  Computation 

One  of  the  primary  and  definable  symptoms  of  the  environmental  conditions  addressed  in 
Section  3.2.2  is  the  actual  sound  path  from  the  target  to  Ownship.  For  purposes  of  this  analysis, 
it  can  produce  not  only  non-measurement  error  adjustment  by  defining  the  differences  between  a 
straight  line  and  actual  sound  path  between  target  and  Ownship,  but  more  importantly,  the 
potential  effects  of  multi-path  conditions  on  the  actual  Sweet  Spot  definition.  The  sound  path 
computations  and  related  data  item  injection  into  the  CDB  supports  all  analysis  efforts.  A  tool 
was  developed  which  will  access  the  defined  SVP  (Section  3.2.2)  and  bottom  files  and  extract 
CDB  values  for  Ownship  Depth  and  Target  Depth  and  Range  to  produce  the  following  listed 
data  items  for  all  data  samples  within  the  set  (normally  1  sec  samples): 

•  Optimum  D/E  (strongest  path  (D/E  Truth)) 

•  Optimum  D/E  -  1  (+  D/E  with  next  highest  signal  strength) 

•  Optimum  D/E  -  2  (+  D/E  with  second  highest  signal  strength) 

•  Optimum  D/E  -  3  (-  D/E  with  next  highest  signal  strength) 

•  Optimum  D/E  -  4  (-  D/E  with  second  highest  signal  strength) 

•  Optimum  D/E  Path  Length 

•  Optimum  D/E  -  1  Path  Length 

•  Optimum  D/E  -  2  Path  Length 

•  Optimum  D/E  -  3  Path  Length 

•  Optimum  D/E  -  4  Path  Length 

•  Measured  D/E  (from  Tracker  Data) 

•  Measured  D/E  -  1  (+  D/E  based  upon  measured  D/E  Variance) 

•  Measured  D/E  -  2  (+  D/E  based  upon  measured  D/E  Variance) 

•  Measured  D/E  -  3  (-  D/E  based  upon  measured  D/E  Variance) 

•  Measured  D/E  -  4  (-  D/E  based  upon  measured  D/E  Variance) 

•  Measured  D/E  Path  Length 

•  Measured  D/E  -  1  Path  Length 

•  Measured  D/E  -  2  Path  Length 

•  Measured  D/E  -  3  Path  Length 

•  Measured  D/E  -  4  Path  Length 

These  data  items  are  placed  into  the  Tracker  data  file  for  easy  access  during  the  analysis 
process. 
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The  sound  path  tool  (Refer  to  Appendix  E)  was  developed  and  has  the  capability  to  calculate 
several  things  when  given  an  input  of  time,  Ownship  position  and  depth,  target  depth  and  range, 
true  bearing  of  target,  target  position  and  measured  D/E  angle  to  target.  A  CDB  output  tool  will 
read  the  CDB  to  provide  the  following  inputs:  historical  SVP  at  both  the  Ownship  and  target 
positions,  historical  bottom  depth  at  Ownship  and  target  and  user-definable  linear  increments 
between  each,  calculated  bottom  slope,  a  forward  ray  trace  defining  path  length  at  user  definable 
increments  +/-  the  measured  WAA  D/E,  and  Eigenray  trace  at  the  truth-specified  target  range 
sorted  by  signal  strength,  also  providing  path  length. 

The  output  of  the  sound  path  tool  is  read  in  by  the  CDB  and  each  piece  of  data  for  every 
sample  that  was  given  to  the  tool  is  stored  with  the  correct  event,  run,  contact  group  and  WAA 
tracker.  If  an  increment  of  anything  greater  than  1  second  is  used,  the  CDB  will  duplicate  the 
data  until  the  next  time  increment  is  reached.  This  method  prevents  data  gaps  from  becoming  a 
problem.  When  measured  data  for  SVP  or  bottom  depth  is  available,  the  CDB  will  use  that  value 
instead  of  historical,  or  it  may  be  overridden.  A  flag  next  to  each  section  identifies  the  data  as 
historical  or  measured. 

While  examining  range  accuracy,  an  analyst  may  suspect  an  error  in  D/E  angle  as  a  potential 
contributor  to  inaccurate  range.  Any  point  on  the  D/E  plot  may  be  chosen,  which  will  allow  the 
analyst  to  access  the  sound  path  data  (such  as  SVP)  associated  with  that  data  point.  The  analyst 
could  also  compare  measured  D/E  with  the  calculated  truth  D/E. 

The  ability  to  call  the  sound  path  tool  is  part  of  the  CDB  interface,  but  is  intended  to  be 
completed  before  the  analysis  of  a  data  set  begins.  The  operator  chooses  the  overall  event,  the 
run  within  that  event,  the  contact  group  of  interest  within  that  run,  and  finally  the  WAA  tracker 
of  interest  that  is  correlated  with  that  contact  group.  Next,  the  user  specifies  a  time  increment  for 
data  output  with  a  minimum  of  1  second.  An  input  file  is  generated  and  the  sound  path  tool  is 
launched.  Processing  a  very  long  run  or  a  fine  time  increment  may  take  minutes  or  hours,  which 
is  why  it  is  advantageous  to  run  this  ahead  of  analysis. 

3.1.4  Non-Measurement  Error  Update 

The  Data  Collection  Process  was  responsible  for  injecting  data  element  corrections  produced 
by  the  Sweet  Spot  Analysis  Process  to  remove  non-measurement  errors.  This  responsibility 
includes  the  documentation  and  tracking  of  all  changes  to  the  data  sets  even  when  those  changes 
might  have  been  inserted  by  one  of  the  other  process  areas.  The  actual  steps  taken  for 
identification  and  correction  of  non-measurement  errors  are  presented  in  Figures  3  and  4  and 
were  normally  performed  outside  of  the  Data  Collection  Process.  Once  the  corrections  were 
inserted,  the  runs  were  separately  installed  in  the  CDB  and  cataloged  by  applying  a  run 
modification  number.  At  this  point,  both  the  initial  and  modified  run  content  resides  in  the  CDB. 

3.1.5  Database  Creation 

This  process  area  was  responsible  for  creating  the  database  to  house  all  of  the  data  and  to 
define  the  schema,  query  structure,  sorting  functionality  and  all  other  data  manipulation 
requirements  for  the  database.  The  responsible  personnel  ensured  that  this  relational  database 
was  maintained  and  backed  up  at  regular  intervals. 
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This  process  relied  on  a  database  SME  (to  establish  the  database,  its  structure,  user 
requirements  and  maintenance  plan).  The  SME  was  then  responsible  for  adding,  enhancing  new 
queries  and  sorting  and  making  architecture  alterations  in  the  event  any  major  CDB  change  was 
required.  In  short  the  database  SME  and  the  Database  Creation  Process  were  responsible  for  all 
activities  concerning  the  construction  and  utilization  of  the  CDB. 

3. 1.5.1  Basic  Data  Content  and  Considerations 

Key  information  concerning  the  content  of  the  CDB  is  listed  below.  In  most  cases  the  data 
set  was  supplied  by  the  Data  Collection  Process  and  forwarded  in  predefined  format  and  content. 
These  points  re-iterate  many  of  the  previously  defined  data  considerations  and  add  items  unique 
to  the  CDB. 

1 .  All  data  was  analyzed  and  stored  on  a  hull-by-hull,  recorded  data  exercise  type,  date/time 
of  run  and  software  version  basis.  Most  data  adjustments  were  impacted  by  one  or  more 
of  these  features;  however,  relational  queries  and  other  database  access  requirements 
were  not  restricted  by  this  partitioning. 

2.  An  early  effort,  working  with  Data  Collection,  was  directed  toward  the  creation  of  a 
matrix  which  used  the  recorded  data  items  as  clues  to  decipher  potential  non¬ 
measurement  errors.  For  example,  the  system  SVP  indicates  an  incorrect  LSS  being  used 
when  the  Ownship  X,  Y,  Z  position  is  identical  to  the  Sensor  X,  Y,  Z,  etc. 

3.  The  tracker  data  sets  defined  include  the  following  data  elements: 

a.  Time 

b.  Bearing  (True  and  Relative) 

c.  Bearing  Rate 

d.  Range 

e.  Range  Rate 

f.  D/E 

g.  SNR  for  Dual  Loop,  FM,  MA  and  D/E 

h.  Sensor  X,  Y,  Z  Position 

i.  Track  Quality  Indicator  (TQI) 

j.  Tracker  Mode  (Manual,  Automatic  Track  Following  (ATF),  Global  Tracking  Trigger 
(GTT),  etc.) 

k.  Validity  Status  for  all  Key  Data  Items 

l.  Variance/Standard  Deviation  values  for  bearing,  range,  D/E  and  SNR 

m.  Other  items  may  become  necessary  as  the  analysis  proceeds. 

4.  The  data  in  the  CDB  also  includes  the  following  information: 

a.  Time-tagged  contact  and  Ownship  truth  position  (presented  in  both  range/bearing  and 
Lat/Long)  as  defined  by  the  operating  range  measurements,  or  in  the  case  of  Open 
Ocean,  the  values  used  for  exercise  reconstruction. 

b.  Time-tagged  truth  and  measured  Ownship  depth  and  recorded  sensor  X,  Y,  Z  position 
(sensor  X,  Y,  Z  will  be  included  in  the  tracker  data  so  the  values  can  be  compared  to 
provide  insight  into  the  status  of  Off-Set  entry  status). 

c.  Time-tagged  contact  and  Ownship  direction  and  velocity  components  (presented  in 
both  Water  and  Ground  Reference). 
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d.  Current  SVP  for  the  operating  area.  Included  in  this  item  will  be  any  recorded  value 
depicting  the  system  selected  LSS  or  designated  as  External  Sound  Speed  (ESS)  for 
AN/BSY-2  based  systems. 

e.  Other  environmental  conditions  including  salinity,  bottom  depth,  background  noise, 
Ownship  radiated  noise  levels,  and  type  and  characteristics  (energy  characteristics 
and  location  installed  on  contact)  of  any  noise  augmentation. 

f.  A  Lat/Long  Operational  Area  Grid  layout  for  each  run 

g.  A  Historical  Bottom  Contour  map  for  the  Operational  Area  cover  by  each  run 

h.  Calculated  D/E  truth  and  pertinent  sound  path  data  associated  with  both  truth  and 
measured  D/E. 

The  above  list  presents  only  the  key  items  and  does  not  reflect  the  CDB  content  in  total. 

3. 1.5.2  CDB  Conceptual  Structure 

This  section  addresses  the  database  implementation  in  a  conceptual  view  only.  Figure  6 
provides  an  overview  of  that  concept.  All  data  contained  in  the  CDB  is  accessible  using  a  top 
down  menu  originating  with  the  exercise.  Each  exercise  which  has  data  runs  housed  in  the  CDB 
is  accessible  and  once  selected,  a  menu  providing  access  to  data  content  levels  ranging  from  the 
entire  exercise  down  to  a  specific  file(s)  within  a  designated  run  or  run  modification  (run  data 
altered  correct  error  or  obtain  compatibility).  Additionally,  applicable  presentation  types  and 
query  tools  are  automated  and  tailored  to  the  level  content  and  made  user-friendly  selectable 
when  working  in  that  tier  of  data.  Some  key  capabilities  included  in  the  data  access  package  are: 

•  Complete  list  of  exercises  and  associated  runs 

•  Top  down  menu  selection  of  data  presentations  at  levels  from  exercise/hull  to 
individual  tracker  or  combinations  of  trackers 

•  Key  high  level  statistic  associated  with  the  runs  listed  (range  of  SNR,  range, 
relative  bearing,  etc.  values) 

•  High  level  charts/graphics,  including  target/O wnship  XY  charting  for  each  run 

•  A  variety  of  analysis-assisting  charts  and  graphs  accentuated  by  defined  Sweet 
Spot  criteria 

•  Ability  to  merge  chart  and  graphs  content  and  to  enhance  interactive  analysis 

•  Calculated  statics  associated  with  the  presented  data  segments  (Value 
Variance/Standard  Deviation,  samples  outside  of  accuracy  threshold,  same 
averaging  interval  to  maximize  data  quality,  etc.) 

•  User  friendly  capability  to  alter  the  Sweet  Spot  criteria  and  apply  them  to 
selected  data  sets 

•  Ability  to  create  and  list  modified  data  files  (all  versions  are  retained  and 
selectable)  that  reflected  need  data  content  changes;  i.e.,  non-measurement  error 
corrections 

•  Production  of  TMA  assessment  output  files  using  the  top  down  menu  (General 
Dynamics  Information  Technology  Lab  and  NUWC) 

•  Composite  statics  including  the  computation  of  Sweet  Spot  Prediction 
Probability  based  upon  selected  Sweet  Spot  criteria  and  qualifying  data 
segments 
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•  Imbedded  Sweet  Spot  interactive  contributors  model  to  create  and  define  the 
complex  dynamics  of  Sweet  Spot  (process  goal,  but  limited  creation  expected 
for  the  initial  analysis  phase) 


The  above  items  are  desired  key  known  characteristics,  but  the  data  structure  and  database 
architecture  are  designed  to  allow  efficient  modification  and  additions  to  these  capabilities. 
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Figure  6.  CDB  Conceptual  Structure 
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The  CDB  accommodates  most  if  not  all  data  elements  that  relate  to  onboard  recorded 
Ownship  Navigation,  Sensor  Data,  Target  and  Ownship  Truth  data,  which  are  the  minimum 
requirement  to  accommodate  WAA  Sweet  Spot  analysis.  This  relational  database  architecture 
has  been  defined  to  have  four  major  levels: 

1.  Events:  Described  as  the  top  level  test  or  exercise  performed  (i.e.,  776  WSAT  and  uses 
UTC  time  reference  for  Start  and  Stop  times  of  the  exercise). 

2.  Runs:  Described  as  specific  maneuvers  within  an  event  (i.e.,  “Run  By”,  Arc  Segment  or  a 
User  Defined  segment  of  a  long  event  defined  by  X-Y  plot  maneuvers  that  would 
compliment  a  WAA  analysis). 

3.  Run  Sections:  Described  as  sequentially  numbered  sections  within  a  run.  This  was 
defined  and  developed  because  of  graphing  limitations  of  3,900  seconds  (1  hr  5  min)  of 
the  WAA  Sweet  Spot  GUI,  however  this  manifested  into  a  means  to  further  delineate 
specific  time  slices  when  narrowing  down  Sweet  Spot  intervals. 

4.  Contact  Groups:  Described  as  1  to  1  matching  of  Target  Truth,  WAA  Sensor,  Reference 
Tracker  (usually  the  Sphere)  and  a  uniform  time  baseline  item  used  to  easily  display  any 
missing  time  elements. 

The  last  operation  of  ETL  (i.e.,  Load),  is  generated  and  performed  based  on  available 
transformed  data  from  the  Raw  Data  extraction.  The  process  remains  consistent  among  varying 
raw  data  sources,  however  data  mapping  must  be  re-developed  for  each  new  heterogeneous  raw 
data  type.  Sample  append  type  queries,  the  basis  for  the  Load  step,  are  available  in  the 
WAASS2k3.mdb  and  are  semi-automated  with  user  entered  parameters  (as  required)  to  catalog 
and  create  new  events.  Sample  update  queries  are  also  provided  which  would  manipulate  CDB 
data  into  their  desired  Runs  and  subsequent  Run  Sections  and  Contact  Group  definitions. 

Once  this  is  accomplished,  preliminary  outputs  are  provided  to  the  Analysis  group  to 
determine  the  following: 

1.  XY  Plots  using  a  team-developed  graphical  tool.  This  provides  a  visual  of  Ownship  vs. 
Target  maneuvers. 

2.  Valid  WAA  Trackers,  their  associated  Reference  tracker,  number  of  targets  and  Contact 
Groupings.  This  may  be  an  iterative  process  and  the  WAA  Sweet  Spot  GUI  allows  for 
temporary  exclusion  of  trackers  to  aid  in  this  activity. 

3.  If  and  how  many  run  sections  need  to  be  created.  Based  on  if  a  Run  is  >  3,900  sec.  in 
duration. 

Through  this  iterative  process,  Contact  Group  definitions  are  the  final  product  and  they  are 
used  to  begin  more  rigorous  Sweet  Spot  analysis. 

The  following  are  outputs  from  the  WAA  Sweet  Spot  database. 

Charts: 

1.  True  Bearing,  Relative  Bearing,  D/E,  SNR  and  Range 
a.  Non-Filtered 

1)  All  included  targets  and  trackers  with  no  Sweet  Spot  initial  criteria  applied 
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2)  Used  for  determination  of  Contact  Groupings 

b.  Filtered 

1)  The  initial  data  points  that  conform  to  the  Sweet  Spot  Initial  Criteria  are  shown 

2)  SNR  >  -12,  Rel  Bng  >15  deg  broadside,  Range  between  .3  and  30  Kyds,  and  D/E 
between  -45  and  20  deg. 

c.  Each  chart  type  can  be  viewed  by  Run,  by  Run  Section  or  by  Contact  Group 

d.  Special  charts  have  been  developed  that  show  Filtered  and  Non-filtered  side  by  side 
to  facilitate  comparisons. 

1)  These  charts  have  timeline  scaling  by  use  of  interactive  sliders  and  are  to  be 
enhanced  to  automatically  generate  new  run  sections  per  contact  group  as  the 
analysis  group  zeros  in  on  potential  Sweet  Spot  time  sections. 

The  following  items  define  additional  outputs  created  and  formatted  by  the  CDB: 

•  Unique  parameters  including  Latitude,  Longitude,  Ownship/Target  Depth, 

Tracker  Measured  D/E  and  Truth  Target  range  and  bearing  are  provided  to  the 
Sound  Path  Evaluation  Tool. 

•  XY  plots  are  generated  by  first  exporting  data  files  in  the  required  format  for  the 
team-developed  tool.  The  capability  is  a  simple  selection  from  the  WAA  Sweet 
Spot  DB  main  menu  bar. 

•  Tactical  Control  System  (TCS)  export  files  are  generated  via  a  main  menu 
selection  and  parameterized  to  select  an  entire  event  or  a  specific  event/run 
combination.  This  is  an  automated  process  which  generates  the  TCS  input  files 
in  the  Automated  Identification  System  (AIS)  defined  file  formats.  A  Sweet 
Spot  flag  field  has  been  designed  into  this  export,  which  is  used  to  identify 
Sweet  Spot  qualified  data  by  the  TMA  algorithms.  Degrees  of  Sweet  Spot  data 
(l=Low  Sweet,  2=Mid  Sweet,  3=High  Sweet)  has  been  discussed  vs.  a  Boolean 
value  of  Sweet  or  Non-Sweet  and  will  be  applied  for  future  analysis.  NUWC 
DB  format  export  is  intrinsically  defined  within  the  Sweet  Spot  CDB  since  the 
NUWC  DB  data  elements  are  a  subset  of  the  Sweet  Spot  CDB.  This  is  a  simple 
set  of  select  queries  that  only  retrieve  NUWC  DB  data  elements  and  insert  them 
into  the  Functional  Analysis  DB  as  a  new  event  with  description  Sweet  Spot 
Modified  plus  the  rest  of  the  original  event  description. 
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3.2  Sweet  Spot  Analysis  Process 

The  Sweet  Spot  analysis  process  begins  with  the  data  review  and  definition  of  non¬ 
measurement  error  conditions.  Figure  7  presents  the  basic  process  for  performing  those  actions. 
Accentually,  a  bias  condition  indicates  a  probable  non-measurement  error  cause.  The  known 
possible  contributors  have  unique  predictable  error  characteristics  over  a  variety  of  operational 
conditions  and  can  thus  be  isolated  given  the  appropriate  data  set  content.  Once  the  error  cause 
is  identified,  the  data  in  most  cases  can  be  adjusted  to  extract  the  presented  error.  The  process  is 
greatly  complicated  when  more  than  one  error  contributor  exists;  however,  procedures  are  being 
developed  to  accommodate  at  least  some  of  these  instances.  When  the  error  cause  cannot  be 
defined,  or  adjustment  is  not  possible,  the  data  will  be  properly  tagged.  All  results  from  this 
process  are  turned  over  to  the  Data  Collection  Process  for  insertion  into  the  CDB. 
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Figure  7.  Non-Measurement  Error  Adjustment  Process 

3.2.1  Non-Measurement  Error  Reduction 

Figure  7  shows  the  process  flow  for  all  data  installed  into  the  CDB  (completed  Data 
Collection  input  processing).  The  non-measurement  errors  generally  fall  into  one  of  three 
categories: 
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•  Hull  Related  -  Likely  to  persist  in  all  data  recorded  from  that  hull  (Array 
Offsets,  etc.) 

•  System  Design  Related  -  Known-design  induced  error  conditions  (straight  line 
Slant  Range  vice  Sound  Path) 

•  Recording/Operator  Related  -  Errors  induced  by  data  recording  techniques  or 
system  operator  error  (incorrect  Ownship/TGT  position,  established  truth  and 
recorded  data  time  difference,  incorrect  operator  entered  LSS  or  SVP,  etc.) 

The  process  first  looked  at  the  first  two  possibilities  and  utilized  known  conditions  for  both 
to  attempt  to  eliminate  any  adverse  contributions  created  by  their  presents.  First,  the  WAA 
Array  Offset  status  was  established  using  the  WAA  Calibration  history,  if  available.  Next,  the 
WAA  Calibration  runs  within  the  data  set  were  evaluated  to  validate  those  findings  or  attempt  to 
define  the  offset  effects  resulting  from  the  WAA  Calibration  deficiencies.  Once  the  array  offset 
status  has  been  resolved,  the  pre-calculated  Sound  Path  parameters  (during  Data  Collection 
Process)  were  used  to  assess  and  adjust  the  WFC  range  to  remove  related  induced  errors.  From 
this  base  any  residual  bias-oriented  error  was  likely  caused  by  one  of  the  recorded/operator 
sources.  Each  has  a  unique  set  of  general  characteristics: 

•  WAA  Applied  LSS  -  Minimum  error  directly  off  the  beam  which  increases 
relative  bearing  moves  in  either  direction  from  the  beam 

•  Recorded  Position  -  Whether  Ownship  or  target,  the  error  is  predictable  and 
adjustable  using  starting  XY  position  data  for  the  offending  unit.  The  problem 
becomes  more  difficult,  but  still  possible,  when  a  position  error  exists  for  both 
Ownship  and  target 

•  Truth/Recorded  Time  -  All  recorded  data  items  indicate  a  lag  or  advance  of 
truth 


3.2.1. 1  Key  Non-Measurement  Error  Considerations 

Listed  below  are  some  key  considerations  applied  during  the  performance  of  the  non¬ 
measurement  error  identification  and  adjustment  actions: 

•  Develop  methods  and  criteria  to  recognize  potential  non-measurement  errors 
and  calculate  the  related  adjustment  values.  There  are  bias  characteristics  that 
will  be  recognizable  and  which  will  signify  which  adjustments  will  need  to  be 
made.  These  characteristics  will  be  defined,  documented  and,  when  possible, 
automated  so  that  the  analysis  can  readily  conclude  which  data  adjustment 
applies. 

•  Separate  data  sets  (exercise  runs)  into  one  of  the  three  following  groups  of 
research  quality:  (1)  Calibration  or  Sensor  Accuracy  runs;  (2)  On  Range  Free 
Play  or  Weapon  Firing  Runs,  and  (3)  Open  Ocean  Runs. 

•  Starting  with  the  Calibration  and  Sensor  Accuracy  Runs,  interrogate  each  run,  to 
determine  the  existence  of  bias  conditions  between  tracker  and  true  range  and 
bearing. 

•  Because  of  the  high  SNR  conditions  of  these  runs,  they  will  likely  be  of 
minimal  value  for  Sweet  Spot  definition  but  will  provide  the  perfect  data  set  for 
fixed  adjustments,  such  as  array  offsets. 
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•  As  all  possible  data  adjustments  are  completed,  assignment  of  a  data  set  quality 
indicator  along  with  other  header  information  and  storage  of  the  modified  data 
set  into  the  database  will  precede  notification  of  the  Sweet  Spot  Analysis  and 
TMA  Solution  Quality  Assessment  Groups  of  new  data  availability. 

•  Apply  applicable  hull-related  fixed  adjustments  derived  from  the  Calibration 
and  Sensor  Accuracy  Run  analysis  to  the  On  Range  Free  Play  or  Weapon  Firing 
and  Open  Ocean  Runs. 

•  Analyze  all  On  Range  Free  Play  or  Weapon  Firing  and  Open  Ocean  Runs  to 
define  further  adjustment  requirements  and  use  the  appropriate  tool  to  adjust  the 
data  accordingly. 

3.2.1.2  Error  Analysis  Process  Description 

The  following  paragraphs  describe  a  possible  end-to-end  error  analysis  procedure  that  could 
be  conducted  with  all  WAA  tracker  data  runs.  All  figures  referred  to  in  this  section  are 
contained  in  Appendix  C.  This  particular  example  was  performed  outside  the  Sweet  Spot 
Database,  using  several  tools  such  as  Excel,  Visual  Basic  and  MATLAB.  While  this  is  not  the 
normal  procedure,  it  is  done  to  illustrate  the  process  before  the  full  Database  capability  was 
available.  This  full  capability  will  be  able  to  perform  all  the  analysis  functions  described  below, 
and  considerably  more  than  that. 

A  couple  of  original  OPEVAL  test  runs  were  analyzed  in  detail  to  see  how  the  effort  would 
proceed  for  all  the  other  runs.  A  run  that  contained  a  rather  large  non-measurement  error 
(reconstruction  error)  was  chosen  to  illustrate  the  point. 

Figure  1  illustrates  all  WAA  and  Spherical  Array  (SB)  trackers  as  originally  recorded  by  the 
on-board  data  collection  system.  The  first  step  was  to  eliminate  all  trackers  for  which  there  is  no 
range  Truth  data  available,  and  limit  the  time  period  of  interest.  This  process  resulted  in  a 
considerable  reduction  of  data  as  depicted  in  Figure  2.  A  bearing  window  of  20  degrees  around 
the  Truth  was  selected  for  this  first  step  in  the  analysis. 

An  approximately  20  minute  run  was  selected  as  illustrated  in  Figure  2.  As  can  be  seen  in 
the  figure,  there  is  a  large  bearing  discrepancy  between  the  WAA  tracker  and  the  reported 
“truth”.  When  available,  we  always  try  to  use  another  non- WAA  tracker  for  comparison  with 
the  WAA  tracker.  A  spherical  array  tracker  is  usually  chosen  for  this  purpose,  considered  as  a 
“reference  tracker,”  because  it  is  the  closest  in  general  characteristics  (such  as  mode  of  operation, 
bandwidth,  etc.)  when  compared  to  WAA  trackers.  The  figure  clearly  illustrates  that  there  is 
very  good  agreement  between  the  WAA  (WR01)  and  spherical  array  (SB02)  trackers,  indicating 
that  they  are  certainly  tracking  the  same  target,  except  for  the  first  and  last  few  minutes  where 
the  SB02  tracker  is  not  tracking  properly.  It  can,  therefore,  be  concluded  that  there  is  a  non- 
WAA  measurement  error  present  in  this  run  due  to  various  possible  conditions  as  outlined 
below. 

Both  trackers  exhibit  a  large  error  when  compared  to  the  Truth.  Figure  3  illustrates  this  error 
for  the  WAA  tracker,  which  reaches  a  peak  value  of  approximately  1 8  degrees  about  two  thirds 
of  the  way  into  this  run.  This  is  certainly  a  huge  error  that  must  be  corrected  prior  to  proceeding 
with  the  analysis. 
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The  first  anomaly  that  can  be  noted  in  the  original  figures  is  that  there  appears  to  be  a 
significant  time  discrepancy  between  the  Truth  and  the  two  trackers.  Examining  the  peak  values 
for  both  traces  in  Figure  2  (truth  peaks  at  12:25:30  and  the  WAA  tracker  at  12:26:12),  it  was 
established  that  the  time  discrepancy  is  approximately  42  seconds,  with  the  trackers  lagging 
behind  the  Truth.  This  error  was  then  corrected  as  depicted  in  Figure  4,  but  this  still  does  not 
solve  the  whole  problem.  When  this  figure  is  carefully  examined,  it  can  be  noted  that  the 
resulting  bearing  error  appears  to  be  a  function  of  true  bearing.  The  nature  of  this  error  is  not 
fully  understood  at  this  time,  but  it  can  be  tentatively  corrected  by  applying  a  simple  linear 
correction  function  of  the  form  Corrected  Bng  =  a  +  b  *  Bng.  Figures  5  and  6  illustrate  this 
result  for  the  WAA  tracker,  indicating  that  the  bearing  error  has  been  essentially  eliminated.  A 
more  sophisticated  correction  function  could  be  used  to  further  reduce  the  bearing  error. 

It  should  be  noted  that  the  above  bearing  error  correction  does  not  directly  affect  the  resulting 
WAA  tracker  range  accuracy.  Only  the  time  misalignment  of  42  seconds  is  significant  in 
considering  the  range  issues  in  further  analysis. 

Figure  7  illustrates  the  true  range  and  the  WAA  range  for  the  time  period  of  interest.  An 
initial  assessment  of  the  range  tracking  quality  indicates  that  the  first  half  of  the  run  is  reasonably 
good,  but  the  second  half  is  certainly  not  nearly  as  good.  We  will  explain  in  the  next  several 
paragraphs  some  possible  reasons  for  this  significantly  large  error  in  measured  WAA  tracker 
range.  Figure  8  shows  the  actual  range  error  based  on  the  measured  and  true  ranges.  Both  of 
these  figures  have  not  yet  been  corrected  for  the  above  mentioned  42  second  time  discrepancy. 

Figures  9  and  10  correct  for  the  time  misalignment.  As  can  be  seen  in  those  figures,  there  is 
a  small  improvement  in  measured  range  accuracy,  but  certainly  not  a  significant  amount.  Other 
factors  are  at  play  here  that  affect  range  accuracy. 

Due  to  the  general  nature  of  WAA  track  processing,  there  are  some  relative  angle  fans  that 
have  poor  range  performance.  To  take  this  factor  into  account,  we  must  look  at  target  relative 
bearing,  as  was  done  and  is  shown  in  Figure  1 1.  In  general,  as  the  tracker  gets  closer  to  forward 
and  aft  end  fire  regions,  its  performance  will  decrease.  This  is  certainly  not  an  abrupt 
performance  decrease,  as  it  happens  gradually.  Figure  12  illustrates  the  resulting  range  errors 
when  poor  relative  performance  bearing  areas  have  been  removed  from  analysis.  As  can  be 
observed  in  this  figure,  the  first  half  of  the  run  is  still  much  better  than  the  second  half.  It  should 
be  noted  that  the  two  halves  of  the  run  represent  the  two  sides  of  the  ship,  starboard  and  port 
respectively. 

Figure  13  depicts  the  percent  error  in  measured  WAA  ranges.  A  straight  line  approximation 
through  the  first  half  of  the  run  results  in  approximately  +4  percent  to  -4  percent.  This  is  a 
reasonably  good  performance  for  the  tracker  under  those  conditions.  The  second  half  of  the  run 
does  not  show  any  beneficial  data. 

Some  possible  reasons  for  this  observed  performance  are  as  follows.  It  is  known  that  no 
acoustic  WAA  calibration  was  performed  on  this  ship.  Lack  of  better  results  during  the  first  half 
of  the  run  is  fairly  consistent  with  this  concept,  where  that  range  accuracy  (or  range  error)  is  a 
function  of  relative  bearing.  Calibrating  the  starboard  side  would  most  likely  result  in  improved 
range  performance  on  that  side.  The  port  side  of  the  ship  has  other  problems.  Lack  of  acoustic 
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calibration  is  certainly  one  of  them,  but  we  also  have  a  significant  decrease  in  tracker  SNR  (not 
illustrated  in  this  analysis),  possibly  due  to  acoustic  propagation  characteristics.  As  can  be  seen 
in  several  figures,  target  range  is  opening  later  in  the  run  which  may  account  for  the  loss  of  direct 
path  propagation,  also  causing  a  significant  decrease  in  SNR. 

Some  of  the  problems  described  in  the  above  paragraphs  are  exactly  the  issues  that  we  must 
study  in  considerable  detail  to  improve  how  the  WAA  system  could  be  better  used  by  the  Fleet. 

3.2.2  Sweet  Spot  Definition  and  Refinement 

The  Sweet  Spot  Analysis  Process  utilizes  the  finalized  and  if  needed,  non-measurement  error 
adjusted  data,  stored  in  the  CDB  to  iteratively  refine  the  established  basic  Sweet  Spot  definition 
by  applying  additional  special  case  contributors  and  attempting  to  define  the  complex 
interactions  which  produce  the  final  Sweet  Spot  as  depicted  in  Figure  8. 


Figure  8.  Sweet  Spot  Analysis  Process 

The  initial  findings  from  the  FY03  NUWCDIVNPT  AN/BSY-2  OPEVAL  data  analysis  and 
this  research  effort  review  of  the  ADM,  WSAT  and  OP/TECH  EVAL  reports  have  established 
that  SNR,  range,  relative  bearing  and,  to  some  extent,  time  in  the  Sweet  Spot  designated  zone, 
are  key  contributors.  In  order  to  reestablish  and  further  define  these  earlier  observations,  a 
thorough  review  and  analysis  of  reports/data  generated  during  these  exercises  was  performed  for 
this  research  effort.  The  primary  technique  applied  for  this  analysis  process  was  to  access 
recorded  data  installed  in  the  CDB  and,  using  current  evolving  Sweet  Spot  characteristics,  query 
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that  data  to  extract  qualifying  data  segments.  The  boundaries  and  quality  of  each  extracted  data 
segment  were  evaluated  and  analyzed  to  identify  causes  for  deviations  from  the  established 
thresholds  (positive  or  negative).  The  data  quality  and  boundaries  assessments  were  directed 
primarily  at  the  WFC  Ranges  but  also  include  bearing  and  to  some  degree  D/E.  The  resulting 
data  set  analysis  findings  redefined  the  query  criteria  and  then,  once  the  query  was  adjusted 
accordingly,  the  process  repeated.  The  data  collected  from  each  iteration  was  formulated  to 
produce  a  statistical  quality  assessment  and  provide  inputs  to  the  Dynamic  Definition  Model  for 
a  thorough  interactive  relations  prediction  definition. 

The  methodology  used  in  this  research  effort  was  focused  upon  which  range  of  Sweet  Spot 
parameters  could  be  identified  using  data  contained  in  reports  from  previous  at-sea  operations. 
Additionally,  to  the  extent  feasible,  interdependencies  between  the  parameters  were  also 
determined.  This  was  accomplished  by  examining  all  available  sea  test  results  from  various 
WAA  configurations  and  submarine  platforms.  Results  examined  included  AN/BSY-2 
(SEAWOLF  Class),  AN/BQG-5  (SSN688  Class),  WAA  ADM  (SSN688  Class),  and  Virginia 
Class  sea  trial  events.  These  events  provided  a  great  deal  of  information  from  which  to  extract 
performance  as  a  function  of  SNR,  relative  bearing  and  range.  The  review  of  report  data  derived 
during  these  exercises  has  confirmed  probable  existence  of  certain  characteristics.  At  the  heart 
of  the  Sweet  Spot  analysis  we  may  be  able  to  predict  the  combination  of  operational  and 
environmental  conditions  that  will  produce  a  rather  small  dynamic  spectrum  where  bearing 
accuracies  of  (.xx)  degrees  and  range  errors  of  under  (x)  percent  can  be  expected.  NOTE:  The 
“xx”  numbers  noted  below  are  classified  (refer  to  Appendix  D  Figure  1 1).  Although  the 
spectrum  associated  with  these  error  conditions  is  relatively  small,  a  slightly  larger  set  of  error 
conditions  of  (.xx)  degrees  in  bearing  and  (x)  percent  in  range  appears  predictable  and  will 
produce  another  level  of  Sweet  Spot  around  the  perimeter  of  the  Sweet  Spot  heart.  Expansion  of 
these  initial  findings  is  the  subject  of  the  continued  analysis  of  additional  at-sea  recorded  data 
from  a  variety  of  exercises. 

3.2.2.1  Results  for  Range  Sweet  Spot: 

As  expected,  results  from  various  sea  trials  and  different  platform/configurations  were 
somewhat  variable.  However,  the  basic  trends  across  all  the  platforms  were  similar.  All  figures 
referenced  in  this  section  are  contained  in  Appendix  D.  Figures  1  and  2  represent  a  composite 
range  accuracy  average  for  all  the  sea  trial  data  examined.  Figure  1  depicts  range  error  vs.  SNR 
for  three  different  ranges  for  a  relative  bearing  near  broadside.  Figure  2  depicts  range  error  at  3 
different  ranges  for  the  high  SNR  case  as  a  function  of  relative  bearing.  It  is  important  to  note 
that  the  plotted  data  is  representative  of  a  WAA  platform  that  has  been  calibrated  to  correct  for 
any  array  misalignment.  From  the  plotted  data,  the  following  initial  Sweet  Spot  definitions  are 
derived: 

Dynamic  Sweet  Spot  Definition  for  WFC  Ranging: 

Target  Relative  Bearing:  For  SNR  XX  and  above,  use  broadside  +/-  XX  degrees 

For  SNR  XX  to  XX,  use  broadside  +/-  XX  degrees 

Target  SNR:  For  SNR  -X  and  above,  use  ranges  out  to  XX  Kyds 
For  SNR  -X  to  -XX,  use  ranges  out  to  XX  Kyds 
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Target  Minimum  Range:  XX  Kyds 

Target  Maximum  Range:  Covered  under  Target  SNR 

The  continuing  effort  focused  upon  enhancing  the  granulation  of  these  boundaries  plus 
incorporating  other  potential  contributors  and  using  a  larger  set  of  recorded  data  that  will  use 
these  values  as  a  baseline. 

3.2.2.2  Results  for  Bearings  Only  Sweet  Spot: 

There  are  occasions  when  the  WFC  range  solution  is  poor  but  the  WAA  tracker  is  still  in 
accurate  solid  track.  These  instances  occur  at  long  ranges  (wavefront  has  little  curvature),  low 
SNRs  that  are  sufficient  for  tracking  but  not  for  WFC  ranging,  and  other  environmental 
conditions  which  preclude  accurate  WFC  ranging.  Figure  3  is  a  plot  of  WAA  bearing  accuracy 
at  or  near  broadside  as  a  function  of  SNR.  Although  there  is  some  deterioration  in  bearing 
accuracy  at  low  SNR,  the  bearing  error,  when  properly  tagged,  can  still  be  very  useful  for 
maintaining  solutions  derived  from  earlier  received  WFC  data  or  even  standalone  solutions  using 
existing  bearings-only  techniques  altered  to  accept  tagging.  Figure  4  is  provided  for  information 
and  shows  that  the  D/E  tracking  loop  can  also  support  tracking  at  low  SNR  levels.  Figure  5  is  a 
plot  of  bearing  error  vs.  relative  bearing  for  the  high  SNR  case.  The  plot  indicates  increasing 
errors  as  the  relative  bearing  nears  +/-  XX  degrees  from  broadside  and  tagging  and  weighting 
will  need  to  be  identified  independently  for  bearing  when  range  tagged  weight  exceeds  a 
predetermined  level.  From  the  data  provided  in  Figures  3,  4,  and  5  the  following  bearings  only 
sweet  spot  definition  is  provided: 

Dynamic  Sweet  Spot  Definition  for  Bearings  Only  Solution: 

Target  Relative  Bearing:  Broadside  +/-  XX  degrees 
Target  SNR:  -XX  and  above 
Target  Minimum  Range:  XX  Kyds 
Target  Maximum  Range:  Not  applicable 

These  conditions  alone  can  define  the  instances  when  Sweet  Spot  is  probable.  What  has  not 
yet  been  defined  are  first,  the  dynamic  interactive  characteristics  of  these  key  contributors,  and 
secondly,  the  effects  additional  lesser  influencing  conditions.  Both  must  be  defined  to  establish 
the  level  of  Sweet  Spot  probability  necessary  and  also  refine  the  boundaries  and  Sweet  Spot  data 
quality  thresholds.  Listed  below  are  some  of  the  suspected  contributors  being  assessed: 

•  Bearing  Rate 

•  Range  Rate 

•  Ownship  Speed 

•  Ocean  Currents 

•  D/E  Accuracy 

•  Multi-path  Vulnerability 

•  Sound  Path,  etc. 
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3.2.2.3  Key  Sweet  Spot  Definition  Considerations 

The  process  used  to  attempt  to  define  the  Sweet  Spot  was  as  follows: 

1 .  The  initial  effort  was  to  define  a  basic  set  of  Sweet  Spot  parameters.  The  recommended 
items  and  associated  values  would  be  something  similar  to  those  defined  in  the  earlier 
phases  of  WAA  data  analysis.  The  Sweet  Spot  Analysis  Team,  as  part  of  their  initial 
tasking,  located  any  existing  tools  and  developed  the  necessary  modifications  for 
calculating  the  sound  path  characteristic  data  for  each  run.  If  the  proper  tools  did  not 
exist,  they  were  created.  The  path  was  calculated  for  all  time  segments  where  a  WAA 
track  existed.  The  data  generated  was  later  used  to  assess  errors  created  by  straight  line 
processing  of  range  data,  vice  sound  path  adjusted,  plus  evaluating  the  existence  and 
degree  of  effect  produced  by  multi-path  conditions. 

2.  After  the  Calibration  and  Sensor  Accuracy  Runs  were  prepared  (properly  adjusted)  by  the 
Data  Collection  and  Database  Creation  Group,  all  runs  were  passed  through  relational 
queries  using  the  attributes  used  to  define  the  Sweet  Spot,  and  any  exceptions  to  the 
declared  Sweet  Spot  bearing  and  range  accuracy  were  extracted.  Working  with  the  Data 
Collection/Database  Creation  and  TMA  Sweet  Spot  Application  Groups,  an  explanation, 
and,  if  appropriate,  resulting  data  adjustments  (on  the  hull-by-hull  basis)  were  made  for 
these  exceptions. 

3.  Once  the  calibration  process  defined  in  Step  2  was  completed,  the  analysis  of  adjusted 
specific  hull  data  provided  by  Range  Free  Play,  Weapon  Firing  or  Open  Ocean  could 
commence.  The  first  step  was  to  perform  a  relational  query  of  the  complete  run  to 
identify  all  segments  meeting  the  attributes  established  for  the  Sweet  Spot. 

4.  Every  extracted  segment  was  analyzed  for  compliance  with  Sweet  Spot  data  quality 
thresholds  and  any  that  failed  were  tagged  and  then  analyzed  to  determine  the  reason  for 
the  failure.  This  process  should  produce  additional  Sweet  Spot  parameters  for 
consideration,  and  possibly  outline  alterations  needed  for  existing  attributes. 

5.  Items  3  and  4  were  an  iterative  process  with  expansion  produced  by  the  addition  of  new 
run  data,  as  it  became  available.  Additionally,  all  results  were  forwarded  to  both  the  Data 
Collection/Database  Creation  and  TMA  Sweet  Spot  Application  Group. 

6.  A  secondary  analysis  was  performed  to  identify  and  quantify  the  extent  of  the  error 
produced  by  certain  operator  and  environment  induced  conditions  known  to  produce 
errors  which  are  currently  deemed  insignificant.  A  couple  of  prime  examples  would  be 
straight  line  vice  sound  path  calculations  for  range,  operator  tendency  to  place  D/E  in 
manual,  etc. 

7.  Based  on  information  received  from  the  TMA  Sweet  Spot  Application  Group,  the  Sweet 
Spot  and  surrounding  area  data  weighting  was  evaluated  and  modified  to  optimize  the 
positive  effects  upon  updated  solution  generation  algorithms. 

3.2.2.4  Sample  of  a  Sweet  Spot  Analysis  using  Fixed  Criteria 

The  following  steps,  in  conjunction  with  the  charts  in  Appendix  D,  provide  insight  into  the 
Sweet  Spot  analysis  process  and  demonstrate  Sweet  Spot  characteristics  for  a  single  data  set. 
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1 .  The  WAA  sensor  is  able  to  process  bearing  and  range  data  over  a  major  portion  of  the 
ship’s  360-degree  azimuth.  However,  to  achieve  the  originally  specified  WAA  sensor 
performance,  the  target  must  be  within  much  more  limited  azimuthal  sectors  that  are 
centered  around  broadside  relative  to  Ownship.  Therefore,  WAA  sensor  data  processed 
outside  those  specified  target  azimuthal  bands  should  be  weighed  much  less  by  TMA 
than  data  collected  within  the  specified  bands  if  the  TMA  solution  development  accuracy 
is  to  be  optimized.  This  strategy  is  not  employed  during  TMA  solution  development 
today. 

2.  Although  WAA  sensor  bearing  and  range  measurements  are  generated  over  a  wide  range 
of  target  SNR,  previous  analysis  of  sea  test  data  has  found  that  WAA  ranging  accuracy 
substantially  degrades  as  SNR  decreases.  Therefore,  sensor  data  collected  outside  a 
minimum  target  SNR  should  be  weighted  much  less  than  data  collected  above  that  SNR 
value  during  the  TMA  solution  development  if  the  solution  range  accuracy  is  to  be 
optimized.  Weighting  of  sensor  SNR  data  is  currently  not  employed  during  TMA 
solution  development  on  tactical  platforms. 

3.  WAA  sensor  ranging  accuracy  is  specified  within  a  defined  range  band.  Many  of  the 
runs  investigated  as  part  of  this  study  contain  range  data  collected  outside  this  range 
band.  Data  collected  and  processed  by  TMA  outside  the  specified  WAA  sensor  range 
band  was  found  to  be  less  accurate  than  that  collected  inside  the  range  band. 

4.  The  following  provides  an  example  of  the  use  of  fixed  criteria  on  a  given  data  set  to 
extract  only  those  range  values  which  fall  within  the  given  criteria.  The  criteria  used  for 
this  particular  example  were  SNR,  range,  and  relative  bearing  and  were  equal  to  the 
following  values: 

a.  SNR  less  than  - 1 5  dB 

b.  Range  between  1.5  and  12  Kyds 

c.  Relative  bearing  30  to  150  degrees  and  210  to  330  degrees 

5.  Figures  6  through  10  in  Appendix  D  provide  an  example  of  the  types  of  information  that 
can  be  extracted  and  analyzed  from  the  database  for  a  given  run.  Figure  6  shows  the 
target  relative  bearing  (truth)  and  the  relative  bearing  as  measured  by  the  WAA  tracker. 
For  most  of  the  run,  the  bearing  is  within  the  criteria  and  the  WAA  tracker  is  tracking  the 
target  very  accurately.  Figure  7  is  a  plot  of  the  SNR  during  the  run.  The  WAA  SNR  (in 
green)  is  within  the  criteria  except  near  the  end  of  the  run.  It  is  at  this  point  that  the 
WAA  relative  bearing  is  approaching  the  WAA  baffles  (bow  and  stem  region)  and  this  is 
most  likely  the  cause  of  the  SNR  drop.  Figure  8  is  a  plot  of  the  tracker  D/E  angle  and 
shows  solid  D/E  track  near  0  degrees  indicating  a  direct  path  arrival.  Figure  9  is  a  plot  of 
the  WAA  measured  range  along  with  range  truth.  For  most  of  the  ran,  there  is  a  10  to  20 
percent  WAA  range  bias  but  the  range  solution  is  solid.  The  range  bias  is  most  likely 
caused  by  small  array  misalignment  and  is  correctable  with  the  insertion  of  calibration 
parameters.  A  calibration  sea  test  which  is  required  on  every  platform  is  required  to 
determine  these  parameters.  Near  the  end  of  the  run,  the  WAA  range  solution  becomes 
very  erratic  most  likely  due  to  the  decreased  SNR  and  the  approach  in  relative  bearing  to 
the  WAA  baffles.  In  current  systems,  all  of  the  WAA  range  values  are  passed  to  the 
TMA  process  with  equal  weighting. 
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Figures  5  through  10  demonstrate  the  effect  of  applying  fixed  criteria  to  the  range 
solution  and  in  this  case  applying  a  simple  yes/no  weighting  algorithm  to  the  solution. 

As  can  be  seen  from  the  early  part  of  the  run,  the  range  dots  are  provided  when  the  fixed 
criteria  are  met.  At  the  end  of  the  run  when  both  the  relative  bearing  and  the  SNR  are 
outside  the  criteria,  the  range  dots  are  given  a  weighting  of  zero  and  are  therefore  not 
plotted  (and  should  not  be  provided  to  TMA).  Since  this  is  the  area  where  the  range 
solution  is  bad,  TMA  solution  accuracy  should  be  improved. 

6.  The  above  simple  example  demonstrates  the  use  of  the  sea  test  database  to  evaluate  the 
use  of  a  simple  set  of  criteria  to  improve  the  range  solution  quality  provided  to  the  TMA 
process.  Future  use  of  the  database  tool  would  include  variations  in  the  criteria  such  as 
the  use  of  dynamic  criteria  associated  with  additional  parameters  such  as  range  rate, 
bearing  rate,  D/E,  etc. 

3.3  TMA  Sweet  Spot  Application  Analysis 

TMA  is  included  in  the  Sweet  Spot  analysis  process  to  demonstrate  that  solution 
improvements  can  result  from  the  increased  bearing  and  range  accuracy  even  if  they  only  exist 
for  brief  periods  of  time  during  a  typical  engagement.  The  initial  TMA  analysis  focused  on  only 
multi-measurement  Parameter  Evaluation  Plot  (PEP),  one  of  the  available  background  solution 
processing  capabilities  (KAST/MLE)  and  MATE,  for  display  and  control  purposes  only. 
Realizing  the  ultimate  Sweet  Spot  TMA  application  approach  would  include  a  scaled  weighting 
scheme,  current  research  resources  were  forced  to  focus  on  a  simple  YES  or  NO  concept.  The 
TMA  Analysis  Process  had  access  to  a  variety  of  CDB  stored  runs  previously  interrogated  and 
tagged  to  denote  whether  the  data  was  in  the  Sweet  Spot  or  not  for  both  bearing  only  and 
bearing/range.  The  TMA  algorithms  would  then  either  use  or  not  use  a  data  item  accordingly. 
Under  these  conditions  the  amount  of  qualifying  data  in  each  run  became  a  factor  in  evaluating 
the  solution  accuracy  improvements.  These  factors  will  continue  to  be  defined  as  the  research 
effort  continues.  Figure  9  details  the  TMA  analysis  process. 
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Figure  9.  TMA  Assessment  Process 

3.3.1  TMA  Sweet  Spot  Application  Key  Points 

The  following  process  has  been  implemented  to  support  the  iterative  nature  of  this  research 
project.  Data  sets  are  received  from  the  Sweet  Spot  Analysis  team.  These  data  sets  have  been 
properly  adjusted  to  remove  the  error  effects  created  by  non-measurement  errors.  These  data 
sets  are  also  properly  formatted  and  made  available  in  both  a  tagged  and  untagged  form.  The 
resulting  TMA  output  and  possible  local  editing  will  produce  Sweet  Spot  enhancement 
recommendations.  These  enhancements  can  then  be  analyzed  and  the  cycle  then  repeated. 

1 .  The  first  step  was  to  define  and  install  a  stable  Combat  Control  TMA  baseline  (TI04). 
Once  installed,  the  baseline  was  thoroughly  tested  and  all  potential  problem  conditions 
including  those  resulting  from  facility  limitations  evaluated  and,  if  necessary,  resolved. 
Baseline  stability  was  crucial  in  order  to  provide  realistic  comparisons  for  “before  and 
after”  demonstrations.  It  was  also  vital  to  rule  out  any  installation  contributions  that 
could  produce  questionable  results  (positive  or  negative)  during  the  Sweet  Spot 
tagged/weighted  data  assessments. 

2.  Working  in  conjunction  with  the  Data  Collection  team,  data  requirements  were  defined  to 
allow  recorded  tracker,  Ownship  and  other  key  run  data  to  be  injected  into  the  Combat 
Control  Subsystem.  Interface  specifications  and  associated  software  were  developed  and 
installed  to  support  both  General  Dynamics  Information  Technology  and  NUWC  TMA 
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capabilities.  Special  consideration  was  applied  to  time  tag  the  recorded  data  because 
meaningful  assessments  require  total  synchronization  of  all  inputs. 

3.  Much  of  the  recorded  data  available  falls  under  the  basic  categories  of  Calibration  or 
Sensor  Accuracy  Runs.  These  run  types  would  normally  have  minimal  value  due  to  the 
high  SNRs  associated  with  the  run  objectives.  However,  within  these  data  sets  there  exist 
a  variety  of  geometric  conditions  which  served  to  validate  proper  insertion  of  run  data 
and  establish  basic  bearing  and  range  solution  accuracy. 

4.  The  primary  source  of  Sweet  Spot  TMA  assessment  data  came  from  On-Range  Free  Play, 
Weapon  Firing  or  Open  Ocean  data.  The  dynamic  and  realistic  nature  of  these  types  of 
exercises  provides  true  insight  into  the  degree  of  improvement  created  by  the  Sweet  Spot 
tagging.  Unfortunately,  the  selection  of  these  types  of  exercises  contained  in  the  existing 
database  was  limited.  The  number  of  geometric  and  environment  conditions  produced 
did  not  allow  for  a  complete  proof  of  concept;  however,  major  trends  were  defined  and 
future  data  requirements  were  established. 

5.  Establish  characteristics  already  embedded  within  TI04  for  MATE  and  Multi-Measure 
PEP  solutions  to  allow  for  operator  control  of  the  data  sets  to  be  used  by  the  applicable 
algorithms.  Controls  were  developed  to  provide  the  capability  to  select  sample  averaging 
intervals.  This  capability  greatly  improved  the  assessment  efficiency  and  accuracy.  The 
features  reduced  the  need  for  manual  recording  of  data  plus  hardcopy  extraction  of  time- 
tagged  results. 

6.  Sweet  Spot  definition  information  obtained  from  the  Sweet  Spot  Analysis  Process 
assisted  in  the  creation  of  additional  alterations  within  the  data  filtering/averaging  and 
internal  TMA  algorithm  filtering/processing  software  and  therefore  maximized  the 
effective  use  of  the  Sweet  Spot  weighted  data.  This  again  was  an  iterative  process  of 
Sweet  Spot  definition  and  Combat  Control  algorithm  adjustment. 

7.  A  formal  event  using  tagged  and  untagged  data  sets  could  be  conducted  when  the  Sweet 
Spot  parameters  were  finalized.  The  final  TI04  algorithm  adjustments  were  not 
completed  to  allow  for  the  systematic  recognition  of  Sweet  Spot  tagged  data,  and 
therefore,  the  complete  process  was  not  verified  during  this  phase  of  the  research  project. 
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4.0  Summary 

The  focus  of  this  first  year  of  this  research  effort  was  on  data  collection,  data  definition  and 
translation,  database  creation,  the  verification  and  refinement  of  previously  defined  WAA  Sweet 
Spot  parameters,  refinement  of  the  dynamic  Sweet  Spot,  including  additional  contributions,  and 
a  demonstration  of  the  positive  effects  on  TMA  capabilities,  if  possible,  within  the  funding 
constraints  of  this  effort.  The  importance  of  a  well-defined  and  documented  process  for 
analyzing  and  proving  the  Sweet  Spot  criteria  was  crucial  and  we  have  established  a  process  that 
is  repeatable,  predictable  and  can  provide  us  with  the  basis  for  verifiable  results. 

This  Phase  I  effort  has  developed  the  process  of  gathering  and  formatting  at-sea  collected 
data  into  a  CDB,  which  allows  further  research  into  developing  weighting  factors  that 
characterize  the  quality  of  the  WAA  sensor’s  received  data  at  various  locations  within  the 
sensor’s  performance  envelope.  The  process  for  analyzing  and  developing  the  Sweet  Spot 
criteria  has  been  well-defined  and  is  a  documented  process  that  is  repeatable,  predictable  and  can 
provide  basis  for  verifiable  results. 

Gathering  the  at-sea  data  from  TECHEVAL,  OPEVAL  and  other  evolutions  for  submarines 
with  WAA  required  extensive  assistance  from  outside  agencies  and  the  establishment  of  the 
mechanisms  required  to  transport  that  secure  data  to  the  General  Dynamics  Information 
Technology  research  facility.  In  order  to  determine  what  portions  of  that  data  for  various  runs 
would  be  cogent  to  our  analysis  required  a  virtual  bit-by-bit  review  of  recorded  data  files  due  to 
the  wide  array  of  data  formats  from  ship  to  ship  that  had  to  be  transformed  into  our  CDB 
structure.  As  a  result  of  this  research  effort  recorded  data  from  subsequent  TECHEVAL, 
OPEVAL  and  other  evolutions  for  submarines  with  WAA  are  now  standardized  and  installed  in 
our  CDB.  While  not  all  data  is  100  percent  compatible,  methods  have  been  defined  to  maximize 
the  usefulness  of  all  recorded  runs. 

Numerous  special  tools  had  to  be  developed  to  correct  and/or  supplement  errors  and  /or 
shortcomings  within  some  of  the  data  content.  These  included  data  deficiency  corrections  for 
incomplete  or  missing  pertinent  data,  Environmental  Data,  Sound  Path  Correction,  and  Non- 
Measurement  Error  Correction.  A  special  tool,  driven  by  data  extracted  utilizing  capabilities 
implemented  in  the  CDB,  has  been  developed  to  assess  the  affects  of  sound  path  and  related 
affects  caused  by  deviations  in  D/E.  Special  tools  and  techniques  have  been  established  for 
defining  and  correcting  for  what  are  considered  to  be  non-measurement  errors  which  result  from 
conditions  created  by  the  data  recording  process  and  those  which  become  a  factor  because  of  the 
increased  accuracy  of  Sweet  Spot  data  (e.g.,  sound  path  vice  straight  line,  installed  array  offsets, 
etc.). 

A  user-friendly  capability  to  query,  using  a  full  complement  of  relational  statements,  has 
been  designed  into  the  CDB.  This  is  the  primary  method  for  identifying  segments  of  data  which 
meet  defined  interim  Sweet  Spot  criteria  and  provides  a  standardized  method  to  review  and 
analyze  a  large  quantity  of  data  runs  containing  a  variety  of  operating  and  environmental 
conditions.  Additionally,  a  vast  array  of  flexible  and  Sweet  Spot  analysis-driven  chart 
generation  capabilities  have  been  integrated  into  the  CDB  design  with  user-friendly  chart 
manipulation  capabilities,  allowing  the  analyst  to  easily  alter  the  presentation  to  focus  on  a 
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particular  area  of  concern.  The  chart  generation  capabilities  are  supported  by  capabilities  to 
provide  statistical  quantification  of  data  contained  in  the  selected  data  segment  (variance, 
standard  deviation,  average,  etc.)  There  is  an  additional  capability  to  place  selected  data 
presentations  in  a  composite  presentation,  which  greatly  enhances  analysis  efforts. 

The  capability  to  perform  a  TMA  assessment  of  Sweet  Spot  data  has  been  installed  and 
checked  out  at  both  the  General  Dynamics  Information  Technology  Facility  and  at  the  NUWC 
Facility  in  Building  1171.  While  the  capability  to  output  data  in  a  form  compatible  with  both 
facilities  is  embedded  into  the  CDB,  the  capability  to  properly  tag  all  data  to  define  each  data 
item’s  Sweet  Spot  compliance  areas  was  not  fully  implemented. 

While  only  a  preliminary  analysis  of  selected  portions  of  the  available  data  were  able  to  be 
performed  due  to  Phase  I  resource  constraints,  some  new  contributing  factors  have  been 
identified  such  as;  D/E  stability,  rate  of  change  in  bearing  or  range,  individual  array  SNR  (FM, 
MA  and  D/E)  vice  established  single  composite  array  SNR,  etc. 

The  capability  now  exists  to  perform  a  more  detailed  assessment  of  how  Sweet  Spot  data  can 
improve  TMA  performance  with  future  at-sea  data,  and  we  believe  that  additional  data  from  USS 
Virginia  TECH/OPEVAL  as  well  as  from  an  exercise  in  the  Pacific  currently  being  planned  to 
evaluate  onboard  acoustic  sensors  on  USS  Cheyenne,  could  provide  the  additional  data  needed. 
We  believe  that  since  the  process  we  have  developed  is  well  defined  and  verified,  a  quick 
analysis  could  be  completed  to  further  refine  the  Sweet  Spot  criteria  when  this  data  is  available. 
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